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Overview 

The  current  research  program  deals  with  ways  by  which  the  representation  of  battlefield 
information  in  general  and  information  about  uncertainty  in  particular,  may  enhance 
Decision  Making,  Situation  Awareness  (SA)  and  Sensemaking  in  battlespace 
environments  (Brickner  and  Lipshitz,  2004).  Previous  stages  of  research  were  based  on 
observations  by  the  research  team  in  a  company-level  simulation  experiment  and  a 
brigade-level  field  experiment  (Brickner  and  Sadot-Parag,  2005a,b  -  See  Appendix  3). 
The  present  stage  included  observation  and  analysis  of  controlled  experimental  research 
that  took  place  in  the  Israel  Defense  Forces  Battlefield  Laboratory  (BatLab  -  Appendix 
4).  A  Battle  Management  System  (BMS)  was  conceptually  designed  by  PAMAM  and 
developed  for  the  experiment  by  the  BatLab.  Because  of  the  high  complexity  inherent  in 
operational  situations  and  scenarios  it  was  decided  to  perform  this  experiment  within  a 
relatively  restricted  battlefield  and  with  a  low  level  of  command.  Additional,  more 
complex,  missions  may  be  simulated  and  run  in  the  future. 

The  research  literature  on  BMS  Human  Factors  in  general  and  uncertainty  representation 
in  BMS  in  particular,  is  based  primarily  on  qualitative  measures:  Task  analyses, 
questionnaires,  Subject  Matter  Expert  (SME)  interviews,  reports  of  observers,  etc.  (e.g., 
Bisantz,  Kesevadas,  Scott,  Lee,  Basapur,  Bhide  and  Sharma,  2002).  Our  goal,  however, 
was  to  use  experimental  tools  that  will  provide  quantitative  performance  measures.  A 
novel  research  methodology  was  developed  by  PAMAM  in  order  to  cope  with  the 
methodological  problems  associated  with  running  a  controlled  experiment  in  a  complex 
command  and  control  environment.  This  environment  (described  below)  allows  the 
scenario  to  develop  naturally  but  restrains  the  development  of  high  levels  of  variability  by 
"resetting"  the  situation  at  predefined  decision  points.  A  SME  prepared  eight  highly 
detailed  operational  scenarios  that  were  programmed  and  implemented  by  the  BatLab 
(see  Appendix  1  for  an  exemplary  scenario). 

After  a  series  of  pretests  and  adaptations,  16  SMEs  from  the  Israeli  Defense  Forces  (IDF) 
were  randomly  assigned  to  two  groups;  eight  in  the  experimental  uncertainty  group  and 
eight  in  the  control  group.  All  subjects  performed  all  of  the  eight  scenarios:  two  training 


1 


Pemem 


and  six  experimental  scenarios.  The  control  group  performed  the  scenarios  with  standard 
tactical  symbols  on  the  BMS  whereas  for  the  experimental  uncertainty  condition  special 
additions  to  the  tactical  symbols  were  developed  by  PAMAM  (Appendix  2).  The 
experiments  were  run  in  the  BatLab  by  BatLab  staff,  under  the  professional  supervision 
of  PAMAM.  Each  experiment  lasted  approximately  four  hours  and  included  the  two 
training  and  six  experimental  scenarios. 

The  representation  of  uncertainty  did  not  have  an  overall  effect  on  decision  making 
quality  but  had  a  significant  effect  on  seven  out  of  the  30  decision  points  (five  decision 
points  within  each  of  the  six  experimental  scenarios).  At  each  decision  point  subjects 
could  ask  for  additional  information  from  one  or  two  sources,  for  the  "price"  of 
expending  some  time.  Additional  information  had  a  significant  positive  effect  on  decision 
making  performance.  Patterns  of  information  request  were  different  in  the  experimental 
group  than  in  the  control  group.  Large  individual  differences  were  found  between  SMEs 
some  of  whom  demonstrated  high  levels  of  command  and  control  expertise  while  others 
seemed  to  lack  sufficiently  deep  understanding  of  the  relations  between  the  BMS  and 
reality.  There  were  also  large  individual  differences  in  rates  of  additional  information 
requests.  Subjects'  SA  probes,  using  the  Situation  Awareness  Global  Assessment 
Technique  (SAGAT  -  Endsley,  2000)  showed  that,  most  of  the  time,  subjects  were 
capable  of  maintaining  reasonable  good  control  of  both  the  near  vicinity  of  the  blue  force 
and  the  relevant  periphery.  No  clear  differences  in  SA  were  found  between  the 
experimental  and  the  control  group. 

In  spite  of  the  restrictions  imposed  by  the  research  methodology  (i.e.,  subjects  required  to 
make  forced  choice  decisions  and  acquire  additional  information  only  at  pre-defined 
locations),  the  SMEs  expressed  overall  highly  positive  opinions  of  the  experiment,  the 
operational  environment,  scenarios,  decision  points,  additional  information  and  SAGAT 
probes.  Furthermore,  some  of  them  expressed  their  wish  to  use  such  a  system  as  a 
training  environment.  The  representation  of  uncertainty  was  evaluated  to  be  a  useful  idea 
both  in  general  (enhancing  the  understanding  that  the  BMS  information  contains  inherent 
uncertainty)  and  specifically  (representing  known  sources  of  uncertainty).  It  should  be 
noted  that  these  opinions  were  only  partially  reflected  in  the  subjects'  performance. 
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Our  conclusion  is  that  uncertainty  information  (when  available  and  reliable)  should  be 
available  to  commanders  as  an  optional  layer  of  information  on  the  BMS. 

Another  important  lesson  is  that  available  information  was  underused,  resulting  in  sub- 
optimal  decisions.  It  is  not  completely  clear  why  several  subjects  refrained  from  requiring 
available  information.  Obviously,  however,  this  kind  of  behavior  was  counterproductive 
and  ways  of  changing  it  should  be  devised. 

Some  individual  subjects  were  excellent  decision  makers  whereas  several  others  did  quite 
poorly.  A  cognitive  task  analysis  is  recommended  in  order  to  better  the  understanding  of 
the  basis  and  the  sources  of  good  battlefield  decision  making  and  impart  it  with  the  less 
proficient  commanders. 

Some  potential  future  research  issues  are  Outlined  in  the  "Recommendations  and  Future 
research"  section. 
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Introduction 

In  recent  years  the  technology  for  acquiring,  processing  distributing  and  displaying 
battlefield  information  has  been  constantly  developing.  Based,  in  part,  on  the  information 
revolution  in  the  commercial  sector  the  military  strives  to  replace  existing  hierarchical 
information  systems  with  a  network  approach  in  which  all  participants  may  serve  as 
sources  as  well  as  clients  of  information  (Leedom,  2001,  2002,  2004).  One  important 
consequence  of  such  approach  is  a  much  wider  distribution  of  Command,  Control, 
Communication  and  Intelligence  (C3I)  information  devices.  Systems  that  in  the  past  were 
the  sole  property  of  high  level  command  and  major  platforms  (e.g.,  bombers),  are 
gradually  being  deployed  to  low  level  units  and  even  to  single  shooters  (for  example, 
helicopters,  tanks,  infantry  soldiers  -  e.g.,  US  Army  Land  Warrior  Program  -  e.g., 
ARNEWS,  2006).  Theoretically,  the  high  availability  of  information  should  lift  much  of 
the  "fog  of  war".  Nevertheless,  the  growing  complexity  of  the  battlefield  may  outweigh 
improved  means  for  obtaining,  distributing  and  displaying  information,  thereby 
maintaining  high  levels  of  uncertainty  (Brickner  and  Lipshitz,  2004).  The  modem 
battlefield  may  contain  great  numbers  and  varieties  of  objects,  many  of  which  may  be 
highly  dynamic  and  many  of  which  may  be  well  concealed.  Furthermore,  the  very  nature 
of  war  seems  to  be  changing  from  conflicts  between  national  armies  to  conflicts  between 
armies  and  various  types  of  guerilla  organizations  (e.g.  the  Israel  -  Palestinian  conflict, 
US  wars  in  Iraq  and  Afghanistan).  This  type  of  adversary  tends  to  be  small,  agile,  mostly 
concealed  and  with  very  low  signatures;  they  are,  therefore,  very  hard  to  detect  and 
recognize. 

Information  and  uncertainty  in  the  commanders'  "information  world" 

Commanders  plan,  command  and  control  missions  that  take  place  in  the  real  world.  The 
term  "information  world"  (Brickner  and  Lipshitz,  2004)  describes  the  combined 
information  that  is  necessary  for  the  performance  of  these  missions.  The  "world"  can  be 
divided  into  four  dimensions: 

■  The  physical  world  (geography,  topography,  atmosphere). 
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■  Friendly  forces  (commanding  levels,  commanded  forces,  own  unit,  other  units, 
neutral  forces). 

■  Enemy  (sites,  targets,  threats). 

■  Mission  stages  (planning,  mission  performance,  debriefing). 

Brickner  and  Lipshitz  (2004)  suggested  defining  "uncertainty"  as  the  gap  between  reality 
and  the  actual  knowledge  (or  situation  awareness)  of  the  commander.  Several  sources 
may  contribute  to  uncertainty.  This  definition  is  somewhat  similar  to  the  one  used  by 
Matthews  and  Shattuck  (2001)  who  preferred  to  look  at  the  "positive"  side  of  the  same 
coin  and  refer  to  SA  rather  than  to  uncertainty.  Brickner  and  Lipshitz  (2004) 
distinguished  between  "subjective  uncertainty"  that  represents  the  gap  between  reality 
and  its  representation  on  the  BMS  (e.g.,  limited  acquisition  abilities,  inefficient 
distribution,  inadequate  BMS  systems)  and  "subjective  uncertainty"  that  may  be  added 
due  to  limitations  of  the  human  operator  (e.g.,  deficient  expertise,  lack  of  attention). 
(Figure  1). 

The  present  report  is  about  "objective  uncertainty".  It  is  argued  that  the  overall  system 
may  have  some  knowledge  regarding  objective  uncertainty  (e.g.,  how  accurate  are  the 
acquisition  systems?  when  was  the  target  acquired?).  It  may,  therefore,  be  possible  to 
represent  some  of  this  uncertainty  as  additional  BMS  information.  In  the  following 
sections  we  refer  to  "objective  uncertainty"  simply  as  "uncertainty". 
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What  is  actually 
acquired? 


What  is  displayed 
and  how? 


Is  C3I  info 
complete  & 
accurate? 


Was  the  right 
decision  made? 


is  action  carried 
out  successfully? 


Figure  1 :  The  flow  of  information  from  the  world  to  the  C3I  system  and  to  the 
commanders  who  make  decisions  and  take  actions. 


Sorting  uncertainty 

Uncertainty  has  many  definitions,  Lipshitz  and  Strauss  (1997)  listed  14  different 
conceptualizations  of  uncertainty  and  similar  terms  (e.g.,  risk,  ambiguity,  turbulence, 
equivocality,  conflict).  Several  definitions  refer  to  the  way  decision  makers  conceptualize 
uncertainty  (e.g.,  Anderson,  Deane,  Hammond  and  McClelland,  1981;  Lipshitz  and 
Strauss,  1997).  Brickner  and  Lipshitz  (2004)  proposed  to  sort  it  along  three  main 
dimensions:  Sorting  by  source,  by  levels  and  by  dimensions  (of  the  "information  world). 
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Sorting  uncertainty  by  source 

Uncertainty  can  originate  from  the  sources  of  information,  from  the  information  display 
or  from  the  limitations  of  the  user.  The  first  two  categories  were  defined  as  objective 
uncertainty  (referred  to  below  as  "uncertainty");  whereas  the  last  source  was  defined  as 
subjective  uncertainty. 

The  combined  sources  of  information  that  feed  C3I  information  system  or  BMS  are, 
inevitably,  partial  and  accuracy-limited.  Modem  technology  enables  fairly  precise 
representations  of  friendly  forces  (e.g.,  by  distributing  their  GPS  readings).  Hostile 
forces,  however,  are  much  more  difficult  to  detect,  recognize  and  understand  and  are, 
therefore,  the  main  contributor  to  BMS  uncertainty. 

A  small  C  I  information  system  display  is  intended  to  represent  events  that  take  place  in  a 
large,  dynamic  and  highly  complex  real  world.  Limited  display  resources  contribute  to 
uncertainty  (e.g.,  the  scale  is  several  orders  of  magnitude  smaller;  height  /  elevation  is 
represented  indirectly,  etc.). 

Commanders  are  affected  by  many  more  subjective  factors  that  may  degrade  their 
decision-making  beyond  the  level  imposed  by  objective  uncertainty  (e.g.,  stress, 
inexperience,  lack  of  attention  resources). 

Sorting  uncertainty  by  levels  of  SA 

Uncertainty  can  be  sorted  according  to  the  three  levels  of  Situation  Awareness  (SA), 
perception,  comprehension  and  projection  (Endsley,  1995  -  see  next  Section  for 
definition).  Generally,  C3I  information  systems  may  be  quite  effective  in  representing 
perception  (i.e.,  detect  and  recognize  major  features  of  relevant  objects)  but  less  so  in 
representing  comprehension  (i.e.,  significance,  intentions,  and  integrative  aspects  of 
relevant  objects)  and  projection  level  (i.e.,  what  will  the  objects  do  next,  where  will  they 
be?). 

Sorting  by  dimensions  (of  the  "information  world") 

Uncertainty  may  pertain  to  any  dimension  of  the  "information  world":  the  physical  world 
(e.g.,  inappropriate  representation  of  heights,  outdated  maps  and  photographs,  lack  of 
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information  on  navigability),  friendly  forces  (e.g.,  their  exact  position  and  condition)  and 
enemy  forces.  Obviously,  uncertainty  with  regard  to  enemy  forces  may  be  dominant. 

Theoretical  background 

Recent  integrative  theoretical  concepts  may  be  relevant  to  the  research  of  uncertainty  in 
C  I  information  systems:  Situation  Awareness  (SA),  Sensemaking  and  Naturalistic 
(Naturalistic)  Decision  Making  (NDM). 

Situation  Awareness  (SA) 

Endsley's  (1995)  definition  of  SA  ("the  perception  of  the  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning  and  the 
projection  of  their  status  in  the  near  future")  is  useful  in  conceptualizing  some  of  the  main 
cognitive  processes  that  are  active  during  the  performance  of  C3I  tasks.  Recent  research 
on  team  SA  (e.g.,  Cooke,  Stout  and  Salas,  2001)  expands  the  applicability  of  the  concept. 
As  stated  above,  we  propose  to  sort  uncertainty  by  three  levels  of  SA:  perception, 
comprehension  and  projection. 

Sensemaking 

The  concept  of  Sensemaking  (Leedom,  2001,  2002,  2004;  Klein,  Phillips,  Rail  and 
Peluso,  2004;  Klein,  Snowden,  Chew  Lock  Pin  and  Ann,  2007)  proposes  a  wider  and 
more  comprehensive  view  than  SA  of  the  cognitive  process  in  individuals,  teams  and 
organizations.  Klein,  et  al.  (2004)  proposed  a  Data  /  Frame  Theory  of  Sensemaking.  The 
authors  view  Sensemaking  as  a  set  of  processes  that  is  initiated  when  a  person  or  an 
organization  recognizes  the  inadequacy  of  their  current  understanding  of  events. 
Recognition  triggers  processes  that  try  to  resolve  the  inadequacy  by  adapting  either  the 
frame  or  the  data  or  both,  thereby  regaining  a  state  of  equilibrium.  In  addition,  Klein,  et 
al.,  (2004)  outline  a  series  of  Sensemaking  characteristics,  several  of  which  seem  relevant 
to  the  operation  of  C3I  information  systems.  Klein,  et  al.  2007,  conducted  a  pilot  study  of 
techniques  to  improve  military  Sensemaking,  specifically  in  the  area  of  Cognitive 
Precision  or  the  “collection  and  connection  of  the  right  dots”,  i.e.  relevant  data.  The 
study  was  designed  to  identify  obstacles  and  inherent  failures  in  Sensemaking  and 
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cognitive  precision  in  individuals  and  in  groups.  Then  it  assessed  the  effectiveness  of  a 
range  of  interventions  on  Sensemaking  abilities,  specifically  cognitive  precision.  The 
interventions  were  tested  with  seven  groups  of  military  and  intelligence  personnel  in  the 
context  of  two  “garden-path1”  type  scenarios  (scenarios  in  which  initial  "strong,"  signals 
are  highly  misleading  with  regard  to  the  actual  circumstances)  -  a  military  planning  task 
and  an  intelligence  assessment  task,  within  which  were  embedded  "weak"  but  highly 
relevant)  signals  for  detection.  The  experiment  successfully  demonstrated  many  of  the 
obstacles  to  Sensemaking;  and,  more  importantly,  found  that  the  interventions  as  a  whole 
proved  useful  in  amplifying  and  identifying  weak  signals  which  were  otherwise  seldom 
detected. 

Decision  making 

In  the  chain  of  activities  of  individual  and  team  users  of  C3I  systems,  decision  making  is 
the  crucial  link.  Lipshitz,  Klein,  Orasanu  and  Salas  (2001)  reviewed  major  trends  in  the 
history  of  decision  making  research  and  the  development  of  NDM  theories.  For  several 
reasons,  the  NDM  approach  and  methodology  seems  most  appropriate  for  C3I  research. 

Proficient  decision  makers:  NDM  views  the  decision  making  processes  of 
experts  as  a  focal  point  for  research  and  investigation  of  decision  making. 

Field  setting:  Zsambock  (1997)  suggested  that  "NDM  is  the  way  people  use  their 
experience  to  make  decisions  in  a  field  setting". 

Process  orientation:  NDM  models  do  not  attempt  to  predict  which  option  will  be 
implemented,  but  describe  the  cognitive  processes  of  proficient  decision  makers. 

Situation-action  matching  decision  rules:  The  study  of  proficient  decision 
makers  leads  to  modeling  decision  making  as  matching  of  a  decision  to  a 
situation,  rather  than  as  choices  between  alternative  decisions  to  the  same 
situation. 


1  Garden-path  sentences  are  relatively  complex  sentences  that  are  often  misinterpreted  by  readers  or 
listeners  e.g.,  Ferreira,  Christianson  and  Hollingworth,  2004.. 
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Context  bound  informal  modeling:  Proficient  decision  makers  may  be  limited  in 
their  ability  to  use  abstract  formal  models.  NDM  models  depict  what 
information  decision  makers  actually  attend  to  and  which  arguments  they 
actually  use  (Cohen  and  Freeman,  1997). 

Empirical  based  prescriptions:  NDM  derives  its  prescription  from  descriptive 
models  of  expert  performance,  rather  than  on  soundly  based  theories  (which  are 
non-existent  in  the  case  of  C3I  systems). 

Decision  making  in  uncertainty 

Lipshitz  and  Strauss  (1997)  defined  uncertainty,  as  “a  sense  of  doubt  that  blocks  or  delays 
action”.  Using  this  definition,  they  identified  three  principal  forms  of  uncertainty  in 
retrospective  reports  on  decision  making  under  uncertainty:  inadequate  understanding  (a 
sense  of  having  insufficiently  coherent  situation  awareness),  lack  of  information  (a  sense 
of  incomplete,  ambiguous  or  unreliable  information)  and  conflicted  alternatives  (a  sense 
that  available  alternatives  are  insufficiently  differentiated).  In  addition  Lipshitz  and 
Strauss  (1997)  identified  five  principal  strategies  of  coping  with  uncertainty:  reducing 
uncertainty  (e.g.,  by  collecting  additional  information);  assumption-based  reasoning 
(filling  gaps  in  firm  knowledge  by  making  assumptions  that  go  beyond  directly  available 
data);  weighing  pros  and  cons  (of  at  least  two  competing  alternatives);  forestalling 
(developing  an  appropriate  response  or  response  capabilities  to  anticipate  undesirable 
contingencies);  and  suppressing  uncertainty  (e.g.,  by  ignoring  it  or  by  relying  on 
unwarranted  rationalization).  Other  researchers  have  also  proposed  similar  lists  of  coping 
strategies  (e.g.,  Klein,  1998). 

Based  on  their  research  results,  Lipshitz  and  Strauss  proposed  the  RAWFS2  Heuristic 
Hypothesis  that  consists  of  quasi-normative  processes  for  coping  with  uncertainty. 
Decision  makers  begin  by  trying  to  reduce  uncertainty  by  collecting  additional 
information;  if  this  is  not  feasible,  they  use  assumptions  to  fill  gaps  in  understanding; 
they  compare  the  merits  of  competing  alternatives  if  such  alternatives  are  available. 

2  RAWFS  -  Reduction,  Assumption-based  reasoning,  Weighing  pros  and  cons, 
Forestalling,  and  Suppression. 
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Proficient  decision  makers  may  retain  a  back-up  alternative  to  guard  against  undesirable 
contingencies  or  suppression  (denial,  distortion  of  undesirable  information)  may  be  used 
as  a  last  resort.  This  model  is  compatible  with  various  naturalistic  decision-making 
models  (Klein,  1998;  Lipshitz,  Klein,  Orasanu  and  Salas,  in  press). 

One  issue  that  is  not  represented  by  decision-making  models,  including  the  Lipshitz  and 
Straus  (1997)  model,  is  the  level  of  awareness  as  to  uncertainty:  does  the  operator  know 
whether  the  available  /  presented  information  represents  high  or  low  levels  of 
uncertainty?  The  first  four  coping  strategies  of  the  RAWFS  model  seem  to  assume  that 
the  operator  is  aware  of  the  level  of  uncertainty  and  seeks  ways  of  coping  with  it, 
whereas,  the  fifth  strategy  assumes  that  uncertainty  is  somehow  ignored  or  suppressed.  In 
reality,  however,  the  selection  of  coping  strategies  and  the  resulting  decisions  may  be 
affected  by  the  operator’s  awareness  of  uncertainty.  If  an  operator  is  totally  unaware  of 
the  level  of  uncertainty  he  or  she  may  act  as  if  they  had  consciously  resorted  to  the 
suppression  strategy.  If,  however,  the  operator  is  aware  of  uncertainty  he  or  she  may 
choose  to  cope  through  one  of  the  other  strategies.  This  issue  can  be  viewed  in  terms  of 
the  knowledge-driven  approach,  which  argues  that  decision  making  in  general  is 
determined  by  operators’  knowledge  driven  strategies,  namely,  action  arguments  that 
describe  how  decision  makers  manipulate  domain  specific  parameters  in  order  to  achieve 
a  certain  goal  (Lipshitz  and  Rozenbaum,  unpublished  manuscript).  From  a  different 
perspective,  one  may  ask  first,  how  to  enhance  the  awareness  to  existing  uncertainty  and 
secondly,  how  to  reduce  the  gaps  between  actual  and  perceived  quality  of  representations 
(displays)  of  the  real  world. 

Uncertainty  in  information  systems  in  general  and  C3I  in  particular  was  observed  by 
several  researchers.  MacEachren  (1992)  discusses  the  issue  of  map  uncertainty.  More 
recently,  Harrower  (2003)  published  a  review  under  the  title  "Representing  LTncertainty: 
Does  it  Help  People  Make  Better  Decisions?"  that  surveys  uncertainty  representation  in 
Geographic  Information  Systems  (GIS).  The  author  concludes  that  in  general  the  answer 
to  the  question  is  positive,  however,  that  more  research  is  still  needed.  Thomson,  Hetzler, 
MacEachren,  Gahegan  and  Pavel  (2005)  proposed  a  typology  for  visualizing  uncertainty 
especially  in  the  context  of  intelligence  information. 
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Summers,  Jones  and  Flo  (2005)  proposed  a  visualization  technique  for  C3I  SA 
information  that  includes  uncertainty  representations.  The  system  underwent  subjective 
evaluations  by  SMEs. 

Pfautz,  Farry,  Fouse,  Bisantz  and  Roth  (2006,  2007)  performed  a  comprehensive  study  on 
the  representing  of  information  to  support  C2  decision  making.  The  study  included  a 
literature  survey,  cognitive  task  analysis  and  various  evaluation  phases.  The  study 
developed  and  evaluated  various  techniques  for  visualization  enhancement.  One  of  the 
main  conclusions  of  the  authors  is  that  representing  uncertainty  is  not  enough;  rather,  it  is 
necessary  to  adapt  a  comprehensive  meta-data  and  meta-information  approach.  Pfautz  et 
al.  (2007)  ran  experiments  in  which  they  validated  the  potential  applicability  of  various 
visualization  techniques.  They  reported  that  saturation,  brightness,  and  transparency  all 
map  naturally  to  probability  and  age  (of  the  information)  assessments,  and  are  orderable, 
while  hue  does  not  map  naturally  and  is  not  orderable;  effects  differ  across  individuals; 
and  the  color  difference  of  the  stimulus  to  the  surrounding  background  has  a  significant 
effect  and  can  be  used  for  encoding. 

Bisantz,  Kesevadas,  Scott,  Lee,  Basapur,  Bhide  and  Sharma  (2002)  proposed  a 
comprehensive  method  for  the  joint  operation  representation  of  tactical  information, 
including  the  representation  of  some  important  aspects  of  uncertainty.  Their  positive 
evaluation  of  the  value  of  more  comprehensive  visualization  of  tactical  information  is 
based  on  a  literature  survey  and  on  cognitive  task  analysis. 

John,  Callan,  Proto  and  Holate  (2000),  performed  two  experiments  in  which  Marine 
Corps  commanders  were  presented  with  maps  and  written  descriptions  of  tactical 
situations  with  and  without  uncertainty  representation  of  enemy  intent.  They  concluded 
that  experienced  commanders  were  not  affected  by  the  representation;  less  experienced 
commanders,  however,  were  likely  to  wait  before  acting  when  uncertainty  was  high. 
Kobus,  Proctor,  Bank  and  Holste  (2000)  presented  commanders  with  geographical  and 
tactical  information  and  asked  them  to  prepare  a  battle  plan.  They  found  that  under 
uncertainty,  experienced  commanders  took  more  time  than  novices  to  complete  situation 
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assessment  processes;  however,  given  the  completed  assessment,  the  experienced 
commanders  finished  the  operational  plan  quicker  and  more  efficiently  than  the  novices. 

LeBlank  and  Summers  (2007)  addressed  another  aspect  of  uncertainty  representation. 
One  of  the  difficult  aspects  of  uncertainty  representation  is  clutter;  BMS  systems  tend  to 
become  overloaded.  The  representation  of  tactical  BMS  information  depends  on  map 
scales:  the  higher  the  scale,  the  more  information  has  to  fit  into  the  same  display  area. 
Inevitably,  there  is  a  tradeoff  between  the  feasible  number  of  displayed  entities  and  the 
level  of  detail  of  each  entity.  The  representation  of  uncertainty  may  exacerbate  this 
problem  by  adding  even  more  information  to  be  displayed. 

Preparatory  research  phases 

During  the  first  stages  of  the  present  research  program,  the  research  team  participated  in 
two  large  scale  experiments,  a  Battle  Lab  simulation  experiment  and  a  field  experiment. 

The  results  of  these  experiments  provided  valuable  insights  into  some  of  the  issues  that 
were  in  the  focal  point  of  the  current  research  program.  Important  lessons  were  learned 
about  gaps  between  commanders'  mental  models  and  BMS  information  representation, 
sources  of  workload,  and  individual  and  team  SA.  Lessons  learned  provided  insight  to  the 
preparation  of  the  test-bed  of  the  present  simulation  experiment.  Specifically,  real  sources 
of  uncertainty  in  BMS  were  identified  and  some  of  them  were  used  as  inputs  in  the 
experiment. 

The  simulation  and  field  experiments  and  main  results  are  presented  in  Appendix  3 


Research  Hypotheses 

The  general  research  question  is  whether  the  representation  of  uncertainty  on  a  BMS  may 
enhance  various  aspects  of  performance  and  information  processing. 

■  Can  representation  of  identified  uncertainty  of  battlefield  information  enhance  the 
decision  making  performance  of  commander  in  a  C2  mission?  (Brickner  and  Lipshitz, 
2004) 
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0  Can  it  affect  the  nature  of  coping  strategies?  (Lipshitz  and  Strauss,  1 997) 

■  Will  uncertainty  information  affect  SA  and  if  so  in  what  respects?  (Endsley,  1995) 

■  Will  uncertainty  representation  affect  Sensemaking  in  terms  of  the  Data  /  Frame 
theory?  (Klein  et.  al.,  2004) 

Method 

The  main  objective  of  the  present  research  program  is  to  investigate  the  effects  of  various 
methods  of  explicit  representation  of  objective  uncertainty  on  Decision  Making  processes 
and  strategies,  SA  and  Sensemaking,  in  a  command  and  control  environment.  These 
issues  are  investigated  in  a  simulated  battle  environment.  The  challenge  was  to  create  an 
experimental  setup  that  is  both  realistic  and  well-controlled  and  yields  as  many 
quantitative  performance  results  as  possible.  For  that  purpose  a  unique  experimental 
method  was  developed. 

Overview  of  the  experimental  method 

The  experiment  is  based  on  a  series  of  experimental  scenarios  that  take  place  in  a 
simulated  urban  area  (based  on  an  actual  town).  The  scenarios,  in  general,  represent  a  low 
intensity  battlefield.  The  blue  forces  consist  of  one  company  operating  as  part  of  a 
battalion.  Each  experimental  trial  consists  of  one  main  mission  (e.g.,  move  through  the 
urban  area  to  a  designated  location  and  capture  a  suspected  terrorist,  no  later  than  the 
zero-hour).  The  mission  commander  is  located  in  a  command  post  and  controls  the  battle 
through  the  BMS  only  (Figure  2).  The  commander  receives  operational  orders  and  a 
battle  plan  that  specify  the  goals  of  the  mission,  the  forces  at  his  disposal,  advancement 
routes,  expected  threats,  zero-hour,  etc.  (See  detailed  method  below  and  Appendix  1). 

In  an  open  simulation  environment  the  commander  is  free  to  make  decisions  at  any 
moment.  As  a  result,  scenario  development  is  bound  to  diverge  and  evolve  differently  at 
each  run.  In  such  a  variable  and  unpredictable  environment  it  is  practically  impossible  to 
extract  quantitative  measures.  To  solve  this  difficulty  a  unique  methodology  was 
developed.  The  commanders'  control  over  the  scenario  was  limited  to  "decision  points". 
The  scenario  is  controlled  by  the  computer  until  a  predefined  point  is  reached  in  which 
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the  commander  is  required  to  make  a  decision.  Each  decision  consists  of  two  stages,  first 
the  commander  has  to  decide  whether  or  not  he  wishes  to  receive  more  information  (e.g., 
from  an  umanned  aerial  vehicle  [UAV])  and,  secondly,  he  has  to  select  a  mode  of  action. 
Both  decision  phases  are  forced  choice,  i.e.,  the  commander  has  to  select  between 
predefined  alternatives  (Appendix  1).  Each  decision  generates  a  predefined  result:  the 
request  for  additional  information  may  or  may  no  yield  useful  information  and  the 
selected  action  path  may  lead  to  favorable  or  unfavorable  operational  results.  The  request 
for  additional  information  has  a  price,  e.g.,  the  UAV  make  take  time  to  get  to  the  point  of 
interest  and  provide  the  information.  In  addition,  the  operational  action  decision  has  a 
price:  it  may  result  in  operational  success  (e.g.,  smooth,  eventless  passage)  or  failure 
(e.g.,  casualties,  unnecessary  engagement  with  hostile  forces,  lost  time).  However,  at  the 
end  of  the  decision  point  all  forces  end  up  in  the  same  position,  regardless  of  the  selected 
decisions;  e.g.,  if  the  decision  resulted  in  casualties,  personnel  was  supplemented  by 
reinforcement.  This  somewhat  artificial  technique  prevents  the  buildup  of  variability 
between  subjects  resulting  from  the  selected  decision(s).  Each  scenario  contains  five  such 
decision  points 

The  commander  was  requested  to  complete  the  mission  as  close  as  possible  to  a 
predefined  zero-hour.  Actual  performance  time  was  affected  by  requests  for  additional 
information  (it  took  time  to  receive  the  information)  and  by  the  decision  taken  (usually 
better  decisions  produced  more  rapid  performance). 

The  commander  also  has  several  additional  tasks.  He  is  asked  read  and  respond  to 
communications  (presented  as  written  announcements)  and  also  to  respond  to  various 
events  in  the  scene  (e.g.,  new  threats,  specific  movements  of  uninvolved  forces).  At  two 
or  three  points  in  each  scenario,  the  scenario  is  frozen  the  commander  is  asked  to  answer 
some  SA  questions  based  on  SAGAT  (Endsley,  2000).  At  the  end  of  each  experiment  the 
commander  undergoes  in-depth  debriefing  that,  among  other  objectives  tries  to  capture 
Sensemaking  processes.  (See  next  Sections  for  a  detailed  description). 

The  BMS  represents  outside  world  events  at  various  levels  of  accuracy  and 
comprehensiveness.  SMEs  that  are  not  exposed  to  uncertainty  representation  have  to 
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figure  out  the  real  situation  in  the  conventional  manner.  SMEs  that  are  exposed  to 
uncertainty  representation  receive  some  additional  information  that  may  or  may  not  aid 
them  in  making  the  right  decisions.  The  representation  (yes  /  no)  of  uncertainty  is  the 
main  independent  variable  of  this  experiment. 


Figure  2:  The  user  interface  with  an  open  dialogue  box  (the  User  Interface  is  in  Hebrew). 
In  the  title  bar  the  name  of  the  system  and  of  the  current  scenario  are  displayed.  The  open 
window  on  the  map  provides  selection  of  various  map  scales.  The  status  bar  (bottom) 
presents  (from  right  to  left)  the  current  scenario  time;  zero-hour;  name  and  scale  of  map; 
and  coordinates  of  a  selected  object  or  of  the  center  of  the  screen  (when  no  object  is 
selected). 

Subjects  (SME) 

Sixteen  SMEs  participated  in  the  experiment.  Subjects  were  randomly  assigned  to  the 
experimental  or  the  control  group,  eight  subjects  in  each). 
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Subjects  ranked  from  Captain  to  Lieutenant  Colonel,  age  range  was  26-38.  All  subjects 
were  male  and  had  operational  experience  as  company  commanders.,  two  subjects  were 
presently  reserve  deputy  battalion  commanders  and  two  were  reserve  battalion 
commanders.  Ten  of  the  subjects  were  infantry  commanders  and  the  others  were  tank, 
engineering  or  artillery  commanders. 

Twelve  out  of  the  16  subjects  had  operational  experience  with  BMS  systems.  Seven  of 
them  had  experience  or  acquaintance  with  more  than  one  BMS  system. 

Apparatus 

The  experiment  is  conducted  in  the  BatLab  which  is  the  major  research  facility  of  the 
Israel  Ground  Forces  (see  Appendix  4  for  a  detailed  description). 

The  experimental  setup,  including  the  BMS,  is  not  standard  and  was  developed  especially 
for  the  present  research  program.  The  experimental  scenarios  were  generated  by  the 
scenario  generator  and  ran  in  the  background.  Their  representation  on  the  BMS  was 
presented  to  the  SME  on  a  17"  LCD  monitor. 

The  BMS  represented  the  battlefield  overlayed  on  a  detailed  aerial  photo.  Basic  map 
functions  where  performed  with  the  mouse  and  the  keyboard  (Figure  2). 

®  Scale:  Zoom  in  /  out  provided  scales  in  the  range  of  1 : 1 ,000-1 : 1 0,000. 

•  Enlarge  selected  area:  a  rectangle  could  be  drawn;  the  selected  area  would  then  be 
enlarged  to  full  window  size. 

•  Distance  measurement:  the  distance  between  each  two  points  can  be  measured  on 
the  map. 

®  Information  on  objects:  additional  information  on  objects  (e.g.,  full  identity, 

affiliation,  location  altitude)  can  be  received  by  clicking  on  the  object  with  the  right 
mouse  key. 

Battlefield  information:  Layers  of  battlefield  information  include  the  blue  forces,  red 
forces,  uninvolved  (black)  and  mission  plans. 
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Accuracy  and  uncertainty:  The  comprehensiveness  and  accuracy  of  BMS  information 
was  varied  to  create  realistic  types  and  levels  of  uncertainty. 

Recordings:  Both  voice  and  video  were  recorded  for  debriefing. 

Scenarios 

The  experiment  included  six  experimental  and  two  training  scenarios.  Each  scenario 
lasted  30-40  minutes.  Each  scenario  contained  five,  forced-choice,  decision  points  as 
described  below. 

All  scenarios  take  place  in  a  three  dimensional  (3D)  simulation  of  a  real  town,  built  in  a 
mountainous  area.  Each  scenario  had  a  detailed  script  that  specified  the  chain  of  events 
(Appendix  1).  Usually,  the  termination  of  one  phase  served  as  the  trigger  for  the  onset  of 
the  next  phase.  Appendix  1  presents  a  detailed  example  of  one  of  the  scenario  scripts. 

Missions  include:  capturing  a  suspect;  locating  and  destroying  an  explosive  lab;  rescuing 
a  downed  helicopter  crew;  setting  an  ambush  for  a  terrorist  group;  surveying  an  area  for 
threats  in  preparation  for  another  mission;  and  raiding  the  headquarters  of  a  terrorist 
organization. 

Forces 

During  the  present  stage  of  research  the  blue  force  consisted  of  an  extended  infantry 
company,  supported  by  additional  forces:  snipers,  tanks,  engineering  forces  and  various 
intelligence  acquisition  devices.  The  company  operates  as  part  of  a  battalion.  Red  forces 
included  various  types  of  guerilla  forces:  gunman,  snipers,  anti-tank  unit,  explosive 
discharges,  obstacles,  etc.  Neutral  forces  included  police,  UN,  press  and  media. 
Uninvolved  forces  included  civilians. 

As  far  as  possible,  the  various  entities  are  represented  by  standard  tactical  symbols, 
excluding  the  representation  of  uncertainty  which  is  new  and  unique  (Appendix  2). 

Each  object  of  each  type  has  sets  of  defined  features;  first,  there  are  the  "real"  features  of 
the  object  in  the  world;  secondly,  there  is  its  representation  on  the  BMS  including:  type, 
identification  accuracy,  location  accuracy,  movement  representation,  information  update 
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rates,  length  of  display  without  refresh  (i.e.,  how  long  and  in  what  manner  will 
information  be  displayed  if  it  is  not  updated). 

Most  of  the  time,  the  representation  of  blue  forces  on  the  BMS  corresponds  fairly 
accurately  to  their  actual  states;  sometimes,  however,  it  may  become  inaccurate,  e.g.,  due 
to  GPS  failure.  The  representation  of  red  forces  is  generally  much  less  complete  and 
accurate  and  entails  the  most  uncertainty. 

Events 

Each  scenario  starts  at  the  opening  position  of  the  operational  event.  The  forces  start 
moving  according  to  plan.  All  along  the  scenario  SMEs  have  to  attend  to  various  types  of 
events  as  described  below. 

Communication:  communication  was  simulated  by  written  messages  that  popped-up  in  a 
window.  The  message  could  be  relevant  (e.g.,  indicating  a  forthcoming  threat,  neutral  or 
irrelevant;  SMEs  had  to  read  it  and  close  the  window  with  the  mouse. 

New  information:  SMEs  were  asked  to  respond  to  the  unplanned  events  on  the  BMS 
(e.g.,  the  appearance  of  a  new  threat).  Because  such  events  could  occur  in  the  periphery 
of  the  battlefield,  SME  had  to  zoom  out  occasionally  in  order  to  detect  them. 

Monitoring  neutral  forces:  SMEs  were  required  to  monitor  the  movements  of  neutral 
forces  including  police,  UN  and  media.  Three  perimeters  were  specified  around  the  force, 
near,  medium  and  far  perimeter.  SMEs  had  to  respond  selectively  each  time  one  of  the 
forces  crossed  the  line  between  perimeters.  The  response  was  performed  with  the  mouse 
and  included  an  indication  of  entity  and  type  of  event.  This  too  required  SMEs  to  zoom 
out  occasionally  and  review  the  periphery  of  the  battlefield. 

Decision  points 

When  the  blue  force  reached  a  predefined  point  the  commander  was  faced  with  a 
dilemma  and  was  required  to  make  a  decision.  Each  scenario  contained  five  such  decision 
points. 

Each  decision  point  included  two  phases.  First  the  commander  was  asked  to  decide 
whether  or  not  to  acquire  more  information  (Appendix  1).  Additional  information  could 
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be  received  from  a  UAV,  a  reconnaissance  balloon  or  other  sources  (forced  choice).  The 
main  cost  of  additional  information  was  time  and  the  possible  exposure  of  the  blue  force; 
hence,  the  commander  had  to  evaluate  whether  or  not  he  really  needs  additional 
information  and  to  decide  which  device  had  the  best  potential  to  provide  useful 
information.  In  addition  the  need  for  information  had  to  be  weighed  against  time 
remaining  until  the  zero-hour.  The  acquired  information  (sometimes  useful  and 
sometimes  not)  was  presented  in  a  window. 

Next  the  commander  had  to  make  the  operational  decision  by  selecting  between  three  or 
four  forced-choice  alternatives  (Appendix  1).  The  selected  decision  was  carried  out  and 
its  outcomes  were  presented  in  a  window  and  on  the  BMS  map.  Outcomes  could  be 
favorable  (e.g.,  proceeding  without  further  events)  or  unfavorable  (e.g.,  blue  casualties, 
uninvolved  casualties,  exposure,  time  loss).  At  the  end  of  the  decision  point,  all  forces, 
regardless  of  the  commander's  decisions,  ended  up  in  the  same  position,  except  for  the 
scenario  time  that  was  calculated  individually.  Thus,  if  a  decision  caused  casualties  the 
commander  lost  points  and  was  informed  about  it;  however  his  forces  were  supplemented 
with  reinforcements  in  order  to  eliminate  the  variability  that  would  otherwise  emerge. 
However,  if  he  lost  too  much  time,  he  would  have  to  cope  with  the  consequences  and 
attempt  to  make  up  for  the  lost  time  during  the  rest  of  the  scenario. 

Real  time  and  scenario  time 

In  the  experiment  "scenario  time"  differs  from  "real  time,"  as  explained  below. 

Each  scenario  has  its  time  frame.  It  starts  at  a  given  time  and  is  supposed  to  end  as  close 
as  possible  to  the  zero-hour  or  as  fast  as  possible  (depending  on  instructions  for  each 
scenario).  Current  scenario  time  and  the  zero  time  are  presented  on  the  bottom  right 
comer  of  the  BMS  display  (Figure  2). 

The  scenario  contains  several  phases  that  may  impose  relatively  long  periods  of  time 
without  much  action  (e.g.,  infantry  force  movements,  waiting  for  information  from  the 
UAV,  etc.).  To  prevent  boredom  and  waste  of  time,  scenario,  time  was  accelerated  5 
times  faster  than  real  time.  However,  since  some  events  require  real  time  (primarily, 
decision  making  processes),  the  time  during  these  events  was  returned  to  real  time. 
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During  events  that  are  external  to  the  flow  of  the  scenario  (SAGAT  and  confidence  rating 
-  see  below),  the  scenario  clock  was  stopped. 

Overall,  a  scenario  of  approximately  30  minutes  (real  time)  represented  three  to  four 
hours  of  operational  events. 

The  uncertainty  manipulation 

All  SMEs  in  the  experimental  group  were  exposed  to  an  information  display  that  included 
partial  representations  of  objective  uncertainty.  In  theory,  a  wide  repertoire  of  uncertainty 
dimensions  may  be  used.  In  the  framework  of  the  present  experiment  only  the  following 
categories  of  uncertainty  were  represented  (Appendix  2). 

•  Location 

®  Direction  of  movement 
®  Update  status 
®  Identity 

•  Affiliation 

All  categories  could  be  applied  to  both  red  and  blue  forces,  however,  only  location- 
uncertainty  was  sometimes  applied  to  the  blue  forces. 

Situation  awareness 

SA  is  measured  two  or  three  times  during  each  scenario  using  SAGAT  (Endsley,  2000). 
The  scenario  was  frozen  and  the  SME  was  asked  a  series  of  questions  regarding  the 
situation.  Question  types  correspond  roughly  to  Endsley's  (1995)  SA  levels  of  perception, 
comprehension  and  projections.  Typical  questions  were: 

Perception: 

What  is  the  indicated  object  (indicated  on  a  map  without  legends). 

Which  of  the  following  is  the  explosive  discharge  (among  the  neutrally  indicated  objects 
on  the  map)? 

Comprehension: 
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Which  of  the  following  objects  endangers  your  unit  (on  the  map,  based  on  type  offeree 
and  distance)? 

How  much  time  remains  until  the  zero-hour? 

Projection: 

Which  force  will  be  first  to  reach  the  junction? 

Will  force  A  and  force  B  collide? 

In  addition,  SMEs  were  asked  to  rate  their  confidence  level  for  each  SA  answer  on  a  scale 
of  1  - 10. 

(See  Appendix  1  for  a  detailed  example.) 

Recording  and  Debriefing 

During  the  whole  sequence  of  the  experiment  both  audio  and  visual  displays  and 
responses  were  fully  recorded.  In  addition,  the  SMEs  were  encouraged  to  speak  up, 
explain  what  they  were  doing  and  to  make  comments.  These  comments  were  manuallv 
recorded  by  the  experimenter. 

At  the  end  of  the  experiment  each  SME  underwent  debriefing,  based  on  a  structured 
interview.  In  addition  to  the  prepared  questions,  debriefing  was  based  on  the  notes  of  the 
experimenter  and  supported,  as  necessary,  by  the  recordings.  We  expected  to  draw 
Sensemaking  insights  from  the  debriefing  process. 

Data  retrieval  and  the  log  file 

During  the  experiment  all  SMEs'  responses  were  recorded  in  a  log  file  which  was  then 
arranged  for  processing.  The  main  data  collected  includes  the  following: 

General  information:  | 

Date  and  time;  SME  personal  information;  Assigned  group  (experimental  or  control) 
Current  scenario;  Scenario  time 

SME  s  responses:  for  each  of  the  following  the  type  of  event  (whenever  applicable)  the 
accuracy  and  response  times  are  recorded: 
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Map  manipulations  (pan,  zoom,  etc.);  Responses  to  communication;  Responses  to  general 
events  (appearance  of  a  new  entity,  etc.). 

Responses  during  decision  point:  additional  information  selection,  the  selected  decision 
and  response  times  for  each  action. 

SA  responses:  selected  response,  time  to  respond,  confidence  level. 

Experimental  design  and  procedure 

A  between  subjects  design  was  selected  where  each  SME  is  assigned  to  either  the 
experimental  (with  uncertainty  representation)  or  the  control  group  (without  uncertainty 
representation).  This  design  was  selected  in  order  to  avoid  possible  effects  of  the 
uncertainty  manipulation  on  control  trials. 

Each  SME  received  general  instruction  and  was  then  familiarized  with  displays  and 
controls.  Next  he  performed  two  training  trials  and  six  experimental  trials.  Each  trial  was 
preceded  by  a  specific  operational  order.  During  the  trial  SMEs  could  use  a  tactical 
symbol  chart  (either  with  or  without  uncertainty  representation)  to  prevent  memory 
lapses.  Debriefing  took  place  after  the  completion  of  the  last  scenario. 

Fifteen  minutes  breaks  were  taken  after  the  two  training  scenarios  and  after  the  third 
experimental  trial. 

The  overall  length  of  each  experiment  was  4.5-5  hours,  including  breaks  and  debriefing. 

Lessons  learned  from  pre-tests 

Several  pretests  were  run  during  the  development  phases  of  the  experiment.  The 
following  are  some  examples  of  lessons  learned  from  the  pre-tests: 

1  he  quickening  of  the  clock  was  adapted  before  reaching  the  present  solution  (five  times 
acceleration  for  inactive  periods,  actual  time  for  decision  making,  zero  for  SAGAT  and 
confidence  ratings). 

We  have  learned  that  some  SMEs  tend  to  complete  the  mission  as  rapidly  as  possible, 
even  at  the  cost  of  ignoring  the  advantages  of  getting  additional  information.  To  change 
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this  conduct  we  strongly  emphasize  that  missions  have  to  be  completed  as  close  as 
possible  to  the  zero-hour  (rather  than  as  early  as  possible).  In  addition,  zero-hours  were 
adapted  to  create  a  balanced  time  pressure  that  is  expected  to  affect  requests  for 
additional  information  during  decision  points.  For  example,  a  highly  cautious  strategy, 
asking  for  as  much  information  as  possible  will  result  in  serious  time  pressure  and  may 
cause  missing  of  the  zero-hour.  In  contrast  a  hasty  strategy,  refraining  from  additional 
information  would  have  a  negative  effect  on  operational  decisions  without  any 
operational  benefit. 

Several  events  requiring  SME  response  were  added  in  order  to  keep  the  subject  active 
and  busy  between  decision  points,  these  include  greater  numbers  of  all  types  of 
(secondary  task)  events  described  in  the  previous  Section. 

The  request  to  monitor  neutral  objects'  movement  was  added  when  it  was  realized  that 
subjects  tend  to  stay  in  a  zoom-in  configuration,  focus  on  close  events  and  ignore  the 
periphery. 

The  number  and  nature  of  uncertainty  representations  was  increased,  varied  and  refined, 
including  location  uncertainty  regarding  blue  forces.  In  addition,  the  location  of  symbols 
on  the  BMS  does  not  always  reflect  the  accurate  position  of  objects  in  the  simulated 
"real"  world.  This  discrepancy  is  sometimes  represented  and  sometimes  not  (i.e.,  SMEs 
face  non-represented  uncertainty  during  all  phases  and  in  both  the  control  and  the 
experimental  condition). 

The  nature  and  size  of  uncertainty  representations  underwent  several  changes  and 
adaptations  until  reaching  its  final  form  (Appendix  2). 
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Results 

Six  main  types  of  results  were  analyzed:  (a)  the  effect  of  the  uncertainty  manipulation  on 
overall  decision  making  and  on  individual  decision  points;  (b)  requests  for  additional 
information  and  their  effects  on  decision  making  quality;  (c)  individual  differences;  (d) 
Performance  time  and  the  zero-hour  (e)  effects  of  SA;  (f)  subjective  debriefing 
information. 

The  uncertainty  manipulation 

Overall  results 

Each  scenario  had  five  decision  points,  each  of  which  had  three  alternative  actions  that 
were  ranked  1  (best)  to  3  (worst).  Decision  quality  was  based  on  outcomes  (friendly 
casualties,  civilian  casualties,  time  loss,  etc.).  It  was  hypothesized  that  overall  decision 
making  will  gain  from  the  representation  of  uncertainty.  Table  1  presents  the  results.  For 
the  uncertainty  group  the  second  best  decision  was  the  most  frequent  selection  (91  out  of 
240)  whereas  for  the  control  group  the  decisions  were  more  evenly  distributed.  For  both 
the  best  answer  was  the  least  frequent.  These  results  however,  are  not  significant. 

Table  1:  Frequency  of  best,  medium,  worst  decisions  by  experimental  group 


Experimental 

Decision  quality 

Total 

group 

1  -best 

2 

3 -worst 

Uncertainty  -  N 

70 

91 

79 

240 

percent 

29.1% 

37.9% 

33.0% 

100% 

Control  -  N 

74 

79 

87 

240 

percent 

30.8% 

33.0% 

36.2% 

100% 

Total  -  N 

144 

179 

166 

480 

percent 

30.0% 

37.3% 

34.6% 

100% 

Individual  decision  points 

In  two  out  of  the  30  (6.6%)  decision  points  there  was  no  variability  in  the  answers.  In 
decision  points  five  and  20,  all  subjects  selected  the  best  decision.  Obviously  these  points 
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could  not  differentiate  between  experimental  conditions  (indicating  poor  selection  of 
alternatives  by  the  experimenters). 

In  nine  out  of  the  30  decision  points  there  were  significant  differences  between  groups. 

In  seven  out  of  these  nine  points  the  uncertainty  group  did  better  and  in  two  points  the 
control  group  did  better. 

Each  of  these  decision  points  was  analyzed  in  search  of  explanations;  presently,  however 
we  cannot  offer  a  general  interpretation  of  the  differences. 

Response  times 

The  uncertainty  manipulation  had  no  consistent  effect  on  response  times.  Table  2  presents 
the  data,  arranged  by  request  for  additional  information.  The  differences  are  not 
significant. 

Xable  2:  Average  time  (seconds)  to  complete  a  decision;  with  no  additional  information, 
with  one  source  and  with  two  sources  of  information. 


Experimental  group 

Response  time 

Asking  for  information 

None 

1  source 

2  sources 

Uncertainty  -  seconds 

19.2 

45.2 

71.7 

(N) 

(171) 

(54) 

(15) 

Control  -  seconds 

18.9 

47.5 

58.2 

(N) 

(163) 

(70) 

(7) 

Request  for  information 

At  each  decision  point  the  subject  had  to  decide  whether  to  make  the  decision  without 
additional  information  or  to  request  additional  information  from  one  source  or  from  two 
sources.  Information  from  two  sources  could  be  requested  either  in  advance  or  after 
receiving  the  information  from  the  first  source,  however,  there  were  no  instances  of 
asking  for  both  sources  in  advance.  It  should  be  noted  that  the  information  received  was 
useful  only  in  approximately  75  percent  of  the  cases 
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Frequencies  of  information  requests 

The  first  finding  that  emerges  is  that  subjects  tended  not  to  ask  for  additional  information. 
Out  of  a  total  of  480  decision  events  (30  decision  points  X  16  subjects):  in  334  cases 
(69.6%)  no  information  was  requested;  in  124  cases  (25.8%)  one  item  of  information 
was  requested;  and  in  22  cases  (4.6%)  two  items  were  requested. 

Overall,  the  uncertainty  group  asked  for  somewhat  more  information  than  the  control 
group  (Table  3).  The  difference  is  marginally  significant  (Pearson  Chi-Square=5.239, 
df=2,  p=0.073). 

Iab!e3:  Asking  for  information  frequency  in  the  experimental  and  the  control  group 


Experimental  group 

Asking  for  information 

None 

1  source 

2  sources 

Total 

Uncertainty  group 

171 

54 

15 

240 

percent 

71.3% 

22.5% 

6.3% 

100% 

Control  group 

163 

70 

7 

240 

percent 

67.9% 

29.2% 

2.9% 

100% 

Total 

334 

124 

22 

480 

I  o  investigate  the  hypothesis  that  uncertainty  representation  might  suppress  requests  for 
information,  we  analyzed  the  asking-for-information  results  of  the  uncertainty  group 
only.  Within  the  240  decision  points  of  the  uncertainty  group  we  distinguished  between 
decision  points  in  which  uncertainties  were  actually  represented  (199  cases)  and  those  in 
which  uncertainty  was  not  represented  (41  cases).  The  frequencies  are  presented  in  Table 
4  and  Figure  3. 

The  analysis  shows  that  when  uncertainty  was  not  represented  there  was  a  significantly 
stronger  tendency  to  ask  for  information  (44.8%  versus  25.6%  for  one  or  two  sources 
combined).  The  results  are  highly  significant  (Pearson  Chi-Square=7.743,  df=2,  p=0.03). 
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Table  4:  Asking  for  information  frequency  in  the  experimental  group,  divided  into 
decision  points  with  actual  uncertainty  representation  and  decision  points  without 
uncertainty  representation. 


Existing  Uncertainty 
representation 

Asking  for  information 

Total 

None 

1  source 

2  sources 

Represented  -  count 

148 

38 

13 

199 

percent 

74.4% 

19.1% 

6.5% 

100% 

Not  represented  -  count 

23 

16 

2 

41 

percent 

56.1% 

39.0% 

4.9% 

100% 

Total  -  count 

171 

54 

15 

240 

percent 

71.3% 

22.5% 

8.3% 

100% 

80 
70 
60 

Percent  of  50 
asking  for  4Q 
information 

30 
20 
10 
0 

0— None  1  source  2  sources 


Uncertainty 
s  Represented 
□  Not  represented 


Figure  3:  Asking-for-information  distribution  in  the  experimental  group  divided  into 
decision  points  with  actual  uncertainty  representation  and  decision  points  without 
representation. 

Effects  of  additional  information  on  decision  quality 
The  question  is  whether  or  not  additional  information  improved  decision  quality.  Table  5 

presents  the  distribution  of  information  request  by  decision  quality. 
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Table  5:  The  distribution  of  decision  quality  by  information  requests.  Percents  are 
computed  for  frequency  of  request. 


Information 

Decision  quality 

Total 

request 

1-best 

2 

3 -worst 

None 

88 

126 

120 

334 

percent 

26.3% 

37.7% 

36.0% 

100% 

1  source 

46 

38 

40 

124 

percent 

37.0% 

30.7% 

32.3% 

100% 

2  sources 

10 

6 

6 

22 

percent 

45.4 

27.3% 

27.3% 

100% 

Total 

144 

170 

166 

480 

Note  that  when  no  information  was  requested,  the  best  decision  was  much  less  frequent 
than  medium  and  worst  decisions;  however,  when  one  or  two  sources  of  information 
were  used  the  best  decision  became  the  most  frequent  one.  The  results  are  marginally 
significant  (Pearson  Chi-Square  =  7.737,  df=4,  p=0.1). 

Order  of  trials 

We  checked  whether  the  request  for  information  was  affected  by  trial  order,  i.e.,  was  the 
tendency  to  ask  for  information  reinforced  or  weakened  as  the  experiment  evolved.  Even 
though  there  is  a  significant  order  effect  (Pearson  Chi-Square=2 1.406,  df=10,  p=0.01).  It 
does  not  seem  to  be  related  to  the  mere  order  but  rather  to  the  nature  of  the  two  last 
scenarios.  In  the  fifth  scenario,  it  was  intuitively  clear  that  the  mission  should  be 
accomplished  as  soon  as  possible;  accordingly,  in  only  15  out  of  80  decision  points  was 
additional  information  requested.  The  last  scenario  involved  friendly  fire  incidents  which 
may  have  enhanced  the  tendency  to  ask  for  information  (35  out  of  80  decision  points). 

Individual  differences 

In  general,  there  were  large  individual  differences  in  both  decision  quality  and  tendency 
to  request  additional  information. 
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Individual  Decision  quality 

The  average  overall  frequency  of  the  best  answer  was  30  percent.  Subjects'  performance 
ranged  from  13.3  percent  for  the  weakest  subject  to  60  percent  for  the  strongest  one.  The 
60  percent  success  stands  out  as  unique  because  the  three  next  best  performing  subjects 
selected  the  best  answer  in  only  43.3,  40.0,  and  40.0  percent  of  the  cases.  Overall 
individual  differences  are  marginally  significant  (Pearson  Chi-Square=42.229,  df=30, 

p=0.068). 

Comparison  of  the  two  experimental  groups  did  not  yield  significant  differences  and  does 
not  reveal  any  clear  patterns. 

Individual  Requests  for  information 

Requests  for  additional  information  varied  strongly  and  were  highly  significant  (Pearson 
Chi-Square=  170.042,  df=30,  p=0.0001). 

One  subject  used  almost  all  opportunities  to  get  more  information  and  asked  for  one  or 
two  sources  in  90  percent  of  the  decision  points.  Next  in  frequency  were  the  two  subjects 
who  requested  information  in  53.3  percent  of  the  decision  points.  On  the  other  end  there 
was  one  subject  who  did  not  ask  for  any  information  whatsoever  and  three  other  subjects 
who  asked  for  information  in  only  one  out  of  the  30  decision  points. 

As  indicated  above  more  information  tended  to  yield  better  answers.  Not  surprisingly, 
asking  for  very  little  information  yielded  bad  decisions  in  all  cases.  The  subject  who 
requested  information  in  90  percent  of  the  decision  points  did  quite  well  (40  percent  best 
answers).  However,  both  subjects  who  asked  for  information  53.3  percent  of  the  time 
were  the  poorest  and  the  second  poorest  decision  makers. 

Performance  time  and  the  zero-hour 

Subjects  were  instructed  to  complete  each  mission  as  close  as  possible  to  a  specified 
zero-hour.  Subjects  could  affect  time  only  during  decision  points  by  asking  or  not  asking 
for  additional  information  and  by  the  decision  itself  which  could  result  in  a  shorter  or 
longer  sequence  of  events. 
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The  results  show  that  subjects  completed  most  of  the  missions  before  the  zero  hour.  Out 
of  96  cases  (16  subjects  *  6  scenarios)  there  were  54  instances  of  finishing  early  and  39 
instances  of  finishing  late  (and  3  missing  data  points).  The  overall  average  was  125 
seconds  earlier  than  the  zero-hour. 

Mission  completion  time  is  significantly  correlated  with  rates  of  asking  for  additional 
information  (Pearson  correlation=-0.308;  p=0.003).  Subjects  who  asked  for  little 
additional  information  tended  to  finish  earlier  than  required;  those  who  asked  for  much 
information  tended  to  be  late;  whereas  a  balanced  use  of  additional  information  led  to 
close  approximation  to  the  zero-hour. 

The  correlation  between  scenario  completion  time  and  decision  quality  was  small  and 
marginally  significant  (Pearson  correlational).  189;  p=0.069). 

Being  late  or  early  with  respect  to  the  zero-hour  depended  on  scenario  (Table  6) 

Table  6.  Average  time  deviation  from  the  zero-hour  by  scenario. 


Scenario  No. 

Average  deviation  time 
from  zero-hour  (seconds) 
(plus=early;  minus=late) 

No  of  cases  (3 
missing  cases) 

2 

-44.9 

16 

4 

-146.0 

16 

5 

+425.7 

15 

6 

+303.5 

15 

7 

-412.8 

16 

8 

+683.9 

15 

Overall  Average 

+125.0 

93 

Note:  scenarios  1  and  3  were  used  for  practive. 


Situation  awareness 

All  SAGAT  questions  had  forced-choice  alternative  answers.  In  general,  subjects  did 
quite  well  on  these  questions.  Given  that  each  question  had  three  or  four  alternative 
answers,  the  correct  answer  was  selected  more  than  half  of  the  time  (52.2%). 
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For  the  analysis  the  SAGAT  answers  were  sorted  as  correct  or  incorrect.  Table  7  presents 
the  overall  results  for  the  two  experimental  groups. 

Table  7:  Overall  SA  results  for  the  experimental  and  the  control  groups:  percent  incorrect 
or  correct 


Experimental 

group 

SA  answer 

Total 

O-incorrect 

1 -correct 

Uncertainty  -  N 

130 

150 

280 

percent 

46.4% 

53.6% 

100% 

Control  -  N 

135 

139 

274 

percent 

49.3% 

50.7% 

100% 

Total  -  N 

144 

179 

554 

percent 

47.8% 

52.2% 

100% 

1  he  experimental  group  had  a  higher  rate  of  correct  SA  answers  than  the  control  group. 
However,  this  difference  is  not  statistically  significant. 

From  a  total  of  36  SAGAT  questions  seven  questions  produced  significant  differences 
between  the  two  groups.  In  five  questions  the  uncertainty  group  did  significantly  better 
and  in  two  questions  the  control  group  did  better. 

Learning  curves 

As  explained  in  the  Method  section,  subjects  underwent  training;  including  practice  with 
two  complete  operational  scenarios  and  then  performed  six  experimental  scenarios  in  a 
fixed  order.  The  methodological  concern  of  whether  decision  making  performance  was 
affected  by  order  (i.e.,  was  increasingly  better  performance  observed  in  later  trials?)  was 
checked.  1  he  general  pattern  of  the  results  does  not  reveal  any  clear  order  effect.  Rates  of 
selection  of  better  or  worse  decisions  varied  by  scenario  but  showed  no  consistent 
tendency  of  improvement  or  degradation  over  the  sequence  of  trials. 

The  pattern  of  results  for  each  scenario  was  similar  for  the  uncertainty  and  the  control 
group  (i.e.,  the  relative  frequency  of  best,  medium  and  worst  decisions  was  the  same)  in 
five  out  of  the  six  scenarios. 
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Debriefing 

At  the  end  of  each  experiment  (that  lasted  approximately  4  hours,  each  subject  was  asked 
a  series  of  questions  and  also  encouraged  to  add  comments.  The  following  is  a  summary 
of  some  major  points. 

Realism  of  the  experimental  setup 

Most  subjects  liked  the  experimental  setup  and  assessed  it  as  a  realistic  representation  of 
operational  challenges,  in  spite  of  the  restrictions  imposed  by  the  experimental  setup. 
They  assessed  the  decision  alternatives  as  representing  real  battlefield  dilemmas. 
Furthermore,  several  subjects  indicated  that  such  a  simulation  should  be  used  to  train 
commanders  in  tactical  decision  making. 

The  main  reservations  voiced  by  the  SMEs  with  regard  to  the  setup  regarded  the  level  of 
control  of  the  simulated  forces.  Subjects  indicated  that  company  commanders  must  be  in 
the  field  with  their  forces  and  use  the  BMS  only  as  an  aid.  It  was  therefore  unnatural  for 
company  commanders  to  remotely  control  their  forces  from  a  BMS  system. 

Practically  all  SA  question  were  rated  as  relevant,  i.e.,  they  sampled  information  of  which 
the  commander  should  be  aware  at  these  points. 

Gaps  between  reality  and  BMS  representation 
All  subjects  knew  that  a  BMS  is  not  a  1 : 1  representation  of  reality.  They  indicated 

various  limitations  and  restrictions  that  limit  the  completeness  and  accuracy  of  BMS 
information.  However,  when  asked  what  they  do  about  it,  the  answers  varied  between:  "I 
am  cautious,  I  try  to  get  more  information  by  radio  or  other  sources,,,"  and  "This  is  what  I 
have  and  act  upon. . 

Asking  for  additional  information 

As  indicated  above,  the  request  for  more  information  varied  between  0  and  50  percent  of 
the  decision  points.  Subjects  indicated  that  their  decision  to  ask  for  more  information  was 
based  on  time  consideration  (time  left  to  zero-hour).  When  time  permitted,  subjects  tried 
to  estimate  whether  information  is  necessary  and  whether  or  not  they  expect  it  to  be 
timely  and  useful.  Some  subjects  were  affected  by  early  experience;  if  some  of  their 
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initial  requests  for  information  yielded  no  useful  information,  they  tended  to  avoid 
requesting  information  during  later  scenarios  and  decision  points. 

Can  the  representation  of  uncertainty  help  in  reality? 

All  subjects  that  experienced  the  representation  of  uncertainty  thought  that  the 

representation  of  uncertainty  was  a  good  idea.  The  main  advantage  is  that  it  strengthens 
the  awareness  as  to  the  limitations  of  BMS  information.  On  the  negative  side,  subjects 
indicated  that  additional  information  may  increase  workload  and  exacerbate  display 
clutter.  Therefore,  some  subjects  suggested  that  uncertainty  should  be  displayed  as  a  layer 
that  can  be  turned  on  and  off  upon  request.  Other  subjects  thought  that  the  information 
might  be  necessary  only  for  beginners,  until  they  get  a  strong  handle  on  the  uncertain 
nature  of  BMS. 

It  should  be  noted  that  these  generally  positive  attitudes  are  not  reflected  in  the 
performance  results. 

At  the  end  of  the  experiment  the  uncertainty  representations  were  also  shown  to  subjects 
of  the  control  group.  Most  of  those  subjects  also  thought  that  such  information  could  be 
useful.  They  too  indicated  the  tendency  to  perceive  a  given  display  as  being  an  accurate 
representative  of  reality  and,  hence,  the  need  to  provide  useful  indications  of  objective 
uncertainty  to  retrain  away  from  this  natural  tendency. 
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Discussion 

The  broad  picture  that  emerges  from  the  results  includes  some  clear  cut  findings  and 
some  less  conclusive  suggestions.  The  main  research  hypothesis  was  that  the 
representation  of  uncertainty  on  the  BMS  will  enhance  decision  making  performance: 
these  manipulations  did  not  reach  an  overall  significant  effect.  However,  it  did  work  for 
some  individual  decision  points  and  it  was  strongly  supported  by  SME  statements  during 
debriefing.  The  clearest  findings  regard  the  request  for  additional  information:  using 
available  information  had  significant  effects  on  (improving)  decision  making  quality. 

All  sixteen  SMEs  that  participated  in  the  experiment  were  experienced  Israeli  Ground 
Forces  commanders.  All  of  them  had  experience  as  field  commanders  at  the  company  and 
/  or  brigade  level.  Most  but  not  all  had  some  experience  or  at  least  acquaintance  with 
BMS  systems.  Despite  the  apparently  common  background,  individual  performance 
levels  varied  from  excellent  to  very  poor.  This  variability  seems  to  have  overshadowed 
some  of  the  other  effects. 

The  uncertainty  manipulation 

Overall  results 

Each  of  the  six  experimental  scenarios  included  five  decision  points,  for  a  total  of  30 
points.  In  each  decision  point  the  SME  had  to  make  a  forced  choice  between  three 
alternative  decisions  that  were  ranked  1  (best)  to  3  (worst),  based  on  outcomes  (friendly 
casualties,  civilian  casualties,  time  loss,  etc.).  The  uncertainty  (experimental)  group's 
most  frequent  selection  was  the  second  best  decision,  whereas  the  selection  distribution 
of  the  control  was  somewhat  more  evenly  divided  across  the  second  and  third  best 
alternatives.  These  results  however,  were  not  significant.  In  both  groups  the  best  answer 
was  the  least  frequently  chosen,  indicating  overall  poor  performance. 
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Individual  decision  points 

In  nine  out  of  the  30  decision  points  there  were  significant  differences  between  the 
experimental  and  the  control  group.  In  seven  out  of  the  nine  points  the  uncertainty  group 
did  better  and  in  two  points  the  control  group  did  better.  Given  that  two  decision  points 
did  not  yield  any  variability  (indicating  inadequate  selection  of  alternative  decisions  by 
the  experimenters)  and  that  only  approximately  70  percent  of  the  uncertainty 
representations  provided  useful,  non-misleading  information,  the  significant  positive 
effect  of  the  uncertainty  manipulation  on  seven  decision  points  seems  to  be  meaningful 
and  to  indicate  its  potential  usefulness.  In  all  of  these  points  uncertainty  representation 
provided  useful  information.  In  both  negative  cases  (better  performance  of  the  control 
group)  uncertainty  information  was  useless  or  misleading.  Despite  a  thorough  analysis  of 
each  decision  point  we  were  unable  to  offer  a  more  general  explanation  as  to  why  the 
manipulation  had  such  effect  at  some  decision  points  and  not  at  others. 

Did  uncertainty  representation  affect  decision  making? 

Our  first  research  hypothesis  stated  that  we  expect  uncertainty  representation  to  have  a 

positive  effect  on  decision  making.  The  actual  results  are  mixed;  the  manipulation  did  not 
yield  an  overall  effect  but  had  a  significant  effect  on  some  of  the  individual  decision 
points. 

It  should  be  noted  that  during  the  preparation  of  the  experiment  we  were  careful  to 
provide  balanced  scenarios  in  which  actual  uncertainty  may  (-70%)  or  may  not  (-30%) 
be  represented;  and  when  represented  it  may  or  may  not  have  been  useful.  Obviously, 
these  precautions  must  have  weakened  the  effects  of  uncertainty  representation.  It  may 
therefore  be  concluded  that  uncertainty  representation  may  be  helpful  under  appropriate 
(i.e.,  scenario  event-dependent)  circumstances. 

Response  times 

The  uncertainty  manipulation  had  no  consistent  effect  on  response  times.  This  may  be 
perceived  as  a  positive  outcome,  indicating  the  uncertainty  manipulation  did  not  add 
much  workload.  It  may,  alternatively,  be  indicative  of  the  relatively  little  attention  that 
some  of  the  subjects  allocated  to  the  uncertainty  representation. 
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Request  for  information 

At  each  decision  point  the  subject  had  to  decide  whether  to  make  the  decision  without 
additional  information  or  to  request  additional  information  from  one  source  or  from  two 
sources.  Information  received  was  useful  only  in  approximately  75  percent  of  the  cases 

Information  request  frequency 

In  theory,  subjects  could  be  expected  to  ask  for  as  much  information  as  possible.  In 
practice,  however,  they  made  relatively  very  little  use  of  additional  information.  Out  of  a 
total  of  480  cases  (30  decision  points  X  16  subjects),  in  334  cases  (69.6%)  no 
information  was  requested,  in  124  cases  (25.8%)  one  item  of  information  was  requested 
and  in  22  cases  (4.6%)  two  items  were  requested. 

The  RAWFS  model  (Lipshitz  and  Strauss,  1997),  states  that  when  uncertainty  is 
identified,  decision  makers  try  to  reduce  it  first  by  trying  to  collect  more  information. 
Three  factors  may  have  suppressed  this  expected  behavior. 

®  One  of  the  lessons  learned  from  pre-tests  was  that  subjects  tended  to  complete  the 
mission  as  quickly  as  possible,  refraining  from  additional  information  requests  in 
order  to  save  time.  To  minimize  the  occurrence  of  this  behavior,  subjects  were 
explicitly  required  to  complete  the  mission  as  close  as  possible  to  the  zero-hour  (and 
not  as  fast  as  possible)  which  left  them  sufficient  time  to  acquire  additional 
information  50-70  percent  of  the  time.  In  spite  of  that,  the  tendency  to  finish  as  fast 
as  possible  still  may  have  affected  performance. 

•  Additional  information  was  useful  only  approximately  75  percent  of  the  instances  in 
which  it  was  available.  Some  subjects,  who  were  disappointed  early  on,  developed 
mistrust  in  the  information.  This  behavior  is  not  rational  because  decision  points 
were  independent  of  each  other  and  because  information  may  or  may  not  be  useful 
in  real  life  as  well. 

As  indicated  by  our  subjects  during  debriefing,  even  fairly  experienced  commanders  may 
fail  to  identify  the  gaps  between  real  world  information  and  its  representation  on  the 
BMS,  even  though,  on  a  theoretical  level  they  all  know  that  such  discrepancies  exist. 
Hence,  if  subjects  did  not  identify  uncertainty  they  may  have  been  overconfident,  not 
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realizing  that  they  actually  needed  additional  information.  In  terms  of  RAWFS  and  based 
on  subjects'  debriefing  it  seems  that  subjects  often  employed  one  of  two  alternative 
strategies  (instead  of  asking  for  more  information).  First,  they  used  assumptions  to  fill  in 
identified  gaps  in  information;  in  particular,  uncertainty  representation  served  as  such 
"filler"  (see  next  Section).  Secondly,  some  of  the  subjects  tended  to  suppress  the 
shortcomings  of  available  information  and  act  as  it  the  picture  were  clear  and  certain. 

Information  requests  in  the  experimental  groups 
The  uncertainty  depiction  group  asked  for  less  information  than  the  control  group.  This 

result  is  counter  intuitive  because  it  was  expected  that  uncertainty  representation  should 
foster  awareness  to  uncertainty  and  promote  requests  for  additional  information. 
Furthermore,  during  debriefing  subjects  indicated  improved  awareness  to  uncertainty  as 
one  of  the  advantages  of  uncertainty  representation. 

In  order  to  examine  the  possibility  that  uncertainty  representation  actually  suppressed 
requests  for  additional  information  we  analyzed  information  request  behavior  within  the 
experimental  group,  comparing  decision  points  in  which  uncertainty  information  was 
available  (199  instances  across  the  eight  subjects)  to  decision  points  without  uncertainty 
information  (41  instances  across  the  eight  subjects).  When  uncertainty  was  not 
represented,  subjects  asked  for  significantly  more  additional  information  (44.8%  versus 
25.6%  of  the  cases).  Clearly,  the  display  of  uncertainty  information  interacted  with  the 
tendency  to  ask  for  additional  information.  When  uncertainty  was  actually  represented  it 
suppressed  requests  for  additional  information;  whereas  when  uncertainty  was  not 
represented  (to  the  uncertainty  group)  it  drove  information  requests  up  to  a  higher  level 
than  in  the  control  group. 

Apparently,  uncertainty  representation  does  strengthen  awareness  to  BMS  uncertainty  but 
is  also  errantly  perceived  as  some  sort  of  substitute  for  additional  information;  allowing 
subjects  to  rely  on  Assumption  Based  Reasoning  (the  "A"  in  RAWFS  -  Lipshitz  and 
Strauss,  1997)  rather  than  on  actual  information  .  Hence  when  uncertainty  is  actually 
represented  it  suppresses  information  requests  but  when  it  is  missing  it  augments  the 
tendency  to  request  additional  information. 
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Effects  of  information  on  decision  quality 
The  findings  clearly  support  the  hypothesis  that  additional  information  may  improve 

decision  making.  When  no  information  was  requested,  the  best  decision  was  much  less 
frequent  than  the  medium  and  worst  quality  decisions;  and  when  one  or  two  sources  of 
information  were  used,  the  best  decision  became  the  most  frequent  one.  Clearly, 
refraining  from  additional  information  was  counterproductive.  In  terms  of  RAWFS 
(Lipshitz  and  Strauss,  1997)  refraining  from  requesting  additional  information  was  both 
irrational  and  counterproductive. 

Order  and  type  of  trials 

We  checked  the  hypothesis  that  asking  for  more  information  may  be  positively  or 
negatively  affected  by  order.  The  order  effect  was  found  to  be  significant.  However,  the 
most  plausible  interpretation  of  the  results  seems  related  to  the  nature  of  the  late  scenarios 
rather  than  to  mere  order.  The  fifth  scenario  required  the  rescue  of  a  helicopter  team  who 
performed  an  emergency  landing  in  the  hostile  area.  Even  though  the  zero-hour  was 
specified,  it  was  intuitively  clear  that  this  type  of  mission  should  be  accomplished  as  fast 
as  possible;  accordingly,  only  in  15  out  of  80  decision  point  events  (5  decision  points  X 
16  subjects)  was  additional  information  requested.  In  contrast,  the  last  scenario  involved 
friendly  fire  incidents;  here  subjects  were  more  cautious  and  asked  for  more  information 
at  more  than  twice  the  frequency  of  the  personnel  recovery  scenario  (35  out  of  80 
decision  points). 

Individual  differences 

In  general,  there  were  large  and  significant  individual  differences  in  both  decision  quality 
and  tendency  to  request  additional  information. 

Individual  Decision  quality 

The  average  overall  frequency  of  the  best  answer  was  30  percent.  Subjects'  performance 
ranged  from  13.3  percent  for  the  weakest  subject  to  60  percent  for  the  strongest  one.  The 
better  performers  tended  to  be  more  senior  commanders  (battalion  and  deputy  battalion 
commander)  and  had  more  experience  with  BMS  systems.  These  findings  are  reflect  a 
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good  deal  of  variability  and  are  not  based  on  a  consistent  relation  between  performance 
quality  and  biographical  data. 

One  subject  did  outstandingly  well  and  chose  the  best  decision  in  60  percent  of  the  cases. 
The  next  best  three  subjects  selected  the  best  decision  in  only  43.3,  40.0,  and  40.0  percent 
of  the  cases.  On  the  other  end  of  the  spectrum,  several  subjects  performed  poorly.  The 
four  weakest  subjects  took  the  best  decision  in  only  13.3,  16.7,  16.7  and  20.0  percent  of 
the  cases. 

Comparison  of  individual  differences  in  the  two  experimental  groups  did  not  yield  clear 
patterns  or  significant  differences. 

Individual  Requests  for  information 

Requests  for  additional  information  varied  strongly.  One  subject  asked  for  one  or  two 
sources  of  additional  information  in  90  percent  of  the  decision  points.  Next  in  frequency 
were  two  subjects  who  requested  information  in  53.3  percent  of  the  decision  points.  On 
the  other  end  there  was  one  subject  who  did  not  ask  for  any  information  whatsoever  and 
three  other  subjects  who  asked  for  information  only  once  across  the  30  decision  points. 

As  indicated  above  more  information  tended  to  yield  better  answers.  Not  surprisingly, 
asking  for  very  little  information  yielded  bad  decisions  in  all  cases.  The  subject  who 
requested  information  in  90  percent  of  the  decision  points  did  quite  well  (40  percent  best 
answers).  However,  both  subjects  who  asked  for  information  53.3  percent  of  the  time 
were  the  poorest  and  the  second  poorest  decision  makers.  In  their  cases  frequent  requests 
for  information  seem  to  reflect  lack  of  self  confidence  and  poor  orientation  rather  than 
rational  handling  of  information. 

In  summary,  the  present  research  supports  previous  findings  (e.g.,  Lipshitz  and  Strauss, 
1997)  that  show  the  positive  relations  between  information  and  decision  making.  The 
failure  of  many  SMEs  to  take  advantage  of  available  information  indicates  lack  of 
experience,  lack  of  consciousness  and  deficient  judgment  and  should  be  dealt  with  in 
training  and  at  command  levels.  The  results  of  the  two  subjects  who  acquired  much 
additional  information  but  performed  poorly,  indicates  that  merely  acquiring  information 
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does  not  produce  good  decisions.  Additional  information  must  be  acquired  at  the  right 
time  and  interpreted  correctly. 

Zero-hour  and  mission  completion 

During  pre-tests  it  had  been  realized  that  subjects  tended  to  try  to  complete  the  missions 
as  fast  as  possible.  The  researchers  were  concerned  that  hastiness  might  affect 
information  requests.  It  was  therefore,  decided  to  set  a  zero-hour  for  each  scenario  and 
require  subjects  to  complete  the  mission  as  close  as  possible  to  that  time.  The  zero-hour 
for  each  scenario  was  set  such  that  it  allowed  time  for  information  requests  at 
approximately  50  percent  of  the  decision  points.  The  zero-hour  and  the  current  scenario 
time  were  displayed  in  the  status  bar  at  the  bottom  of  the  BMS  monitor  (Figure  2). 

In  practice,  subjects'  control  of  the  time  was  very  limited  because  most  stages  of  the 
scenario  were  paced  by  the  computer  and  subjects  could  affect  time  only  during  the 
decision  points.  Subjects  had  a  good  idea  of  how  long  they  had  to  wait  until  receiving 
information  from  a  UAV,  a  balloon,  etc.;  however,  they  could  not  know  in  advance,  how 
the  decision  itself  might  affect  time. 

Most  scenarios  (54  out  of  93  cases)  were  completed  before  the  zero-hour  and  average 
deviation  time  was  positive  (i.e.,  early).  The  highest  positive  deviation  (being  early)  was 
1348  seconds  and  the  highest  negative  deviation  (being  late)  was  1 133  seconds.  Asking 
for  more  information  was  significantly  correlated  with  finishing  the  scenario  early  or  late. 
Additional  information  required  time  and  therefore  subjects  who  asked  for  little 
information  tended  to  be  consistently  early,  those  who  asked  for  much  information  tended 
to  be  late  and  those  who  were  well-balanced  finished  relatively  close  to  the  zero  hour. 

Different  scenarios  created  different  patterns  of  time  deviations.  Subjects  interpreted  the 
operational  situation  and  acted  upon  it.  In  the  crashed  helicopter  team  rescue  mission 
(Scenario  8,  Table  6)  subjects  tended  to  ignore  the  zero-hour  and  complete  the  task  as 
quickly  as  possible  by  refraining  from  asking  for  additional  information,  resulting  in  an 
average  of  683.9  seconds  early.  On  a  mission  that  involved  friendly  fire  (Mission  7,  Table 
6)  subjects  tended  to  be  much  more  cautious,  asked  for  much  more  additional  information 
and  ended  the  mission  412.8  second  late  (average). 
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Situation  awareness 

SAGAT  questions  (Endsley,  2000)  addressed  information  in  the  close  vicinity  of  the  blue 
force  and  well  as  peripheral  information  that  was  considered  relevant.  Overall,  subjects 
did  quite  well  and  selected  the  correct  answer  (out  of  three  or  four  alternatives)  52.2 
percent  of  the  time. 

The  experimental  group  had  a  slightly  higher  rate  of  correct  SA  answers  (53.6%)  than  the 
control  group  (50.7%),  However,  these  differences  were  not  significant. 

From  a  total  of  36  SAGAT  questions,  seven  questions  produced  significant  differences 
between  the  two  groups.  In  five  questions  the  uncertainty  group  did  significantly  better 
and  in  two  questions  the  control  group  did  better.  These  differences  were  not  correlated 
with  other  observed  differences. 

Learning  curves 

Based  on  the  relatively  poor  overall  performance  and  on  subjective  impression,  we 
assumed  that  the  subjects  were  not  sufficiently  trained.  (Recall  that  subjects  underwent 
two  full  training  scenarios  and  then  six  experimental  scenarios  in  a  fixed  order). 

Nevertheless,  there  was  no  clear  order  effect  on  decision  making.  Rather,  decision 
making  performance  was  affected  by  type  of  scenario  rather  than  by  their  order  (see 
previous  Section).  The  pattern  of  results  for  each  scenario  was  similar  in  the  uncertainty 
and  the  control  group  (i.e.,  the  relative  frequency  of  best,  medium  and  worst  decisions)  in 
five  out  of  the  six  experimental  scenarios. 

It  seems  that  two  training  scenarios  were  sufficient  for  acquiring  the  necessary 
proficiency  /  technical  skills;  however,  they  were  not  sufficient  for  acquiring  sufficient 
consciousness  to  BMS  limitations.  It  seems  that  much  deeper  and  more  extensive  training 
is  necessary  for  actually  changing  the  SMEs'  mental  model  (see  Section  on  Sensemaking 
below). 

Debriefing 

At  the  end  of  each  experiment  (that  lasted  approximately  4  hours),  each  subject  was 
asked  a  series  of  questions  was  encouraged  to  add  comments. 
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General  appraisal 

In  general,  all  subjects  expressed  positive  appraisal  of  the  experiment.  Most  subjects  were 
highly  busy  officers  who  volunteered  to  participate  in  the  experiment  and  had  difficulties 
finding  a  five  hour  time  slot.  Nevertheless,  once  they  arrived,  they  took  the  time  and 
patience  to  complete  the  experiment,  make  comments  and  participate  in  the  debriefing 
session. 

Realism  of  the  experimental  setup 

As  indicted  in  the  Results  section,  the  experimental  setup,  scenarios  and  tasks  received 
high  "grades"  for  realism,  in  spite  of  the  restrictions  imposed  by  the  experimental  setup 
(forced  choice  decision  points,  no  control  of  the  commanded  force  except  at  decision 
points).  Several  subjects  indicated  that  a  similar  simulation  setup  should  be  used  to  train 
commanders  in  BMS  usage  and  in  tactical  decision  making. 

With  the  development  of  BMS  for  lower  operational  command  levels,  questions 
regarding  the  role  and  the  desired  location  of  company-battalion  commander  have  been 
raised.  Should  these  commanders  head  their  forces  into  the  battlefield?  Should  they  view 
the  world  directly  or  is  it  better  to  get  an  overview  via  the  BMS  system?  (Schmitt  and 
Klein,  1996).  The  present  experiment  was  not  designed  to  answer  these  questions. 
However,  some  of  the  SMEs  felt  that  the  experimental  situation  was  not  natural  / 
appropriate  for  the  company  level.  Subjects  indicated  that  company  commanders  "must" 
be  in  the  field  with  their  forces  and  use  the  BMS  only  as  an  aid.  It  was  therefore  unnatural 
for  company  commanders  to  control  their  forces  indirectly  via  a  BMS  system. 

Gaps  between  reality  and  BMS  representation 
In  the  experiment  we  got  strong  support  for  the  notion  that  commanders  may  not  have 

real  awareness  of  the  gaps  between  BMS  information  and  reality.  On  the  intellectual  level 
all  subjects  knew  that  a  BMS  is  not  a  1 :1  representation  of  reality.  However,  when  asked 
what  they  do  about  it  -  the  answers  varied  between:  "I  am  cautious. "I  try  to  get  more 
information  by  radio  or  other  sources,,,"  and  "This  is  what  I  have  and  act  upon...".  In  the 
experiment,  the  majority  of  decision  points  (across  subjects)  were  treated  as  "This  is  what 
I  have  and  act  upon..."  resulting  in  many  instances  of  suboptimal  decision. 
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Asking  for  additional  information 

As  indicated  above,  at  each  decision  point  (five  per  scenario)  subjects  had  the  option  to 
ask  for  additional  information  from  one  or  from  two  sources  -  prior  to  making  the 
decision.  In  the  experiment,  subjects  used  relatively  little  additional  information,  thereby 
missing  opportunities  to  make  better  decisions.  Some  subjects  indicated  that  for  them 
asking  for  more  information  depended  on  time  consideration  (time  left  to  zero-hour). 
When  they  thought  that  time  permitted,  they  tried  to  assess  whether  information  was 
necessary  and  whether  or  not  it  may  be  timely  and  useful.  In  reality  (see  "Zero-hour  and 
mission  completion"  above) ,  time  was  more  strongly  affected  by  the  quality  of  the 
decision  (good  decisions  usually  saved  time)  than  by  the  time  spent  waiting  for  additional 
information.  Debriefing  indicates,  however,  that  subjects  had  no  clear  awareness  of  this 
complicated  tradeoff. 

Some  subjects  were  affected  by  early  experience;  if  some  of  their  initial  requests  for 
information  yielded  no  useful  information  they  tended  to  do  without  information  during 
later  scenarios  and  decision  points.  Even  though  decision  points  were  independent  of 
each  other,  these  subjects  considered  this  to  be  rational  behavior.  This  seems  to  be  a  clear 
example  of  "Assumption  Based  Reasoning"  (Lipshitz  and  Strauss,  1997). 

Some  of  the  more  experienced  subjects  actually  tried  to  assess  the  expected  usefulness  of 
the  information  source  based  on  tactical  and  topographical  information  (e.g.,  Could  the 
UAV  see  into  the  alley?).  This  expectation  of  utility  affected  the  decision  to  request 
additional  information. 

Can  the  representation  of  uncertainty  help  in  reality? 

All  subjects  of  the  experimental  group  who  experienced  the  representation  of  uncertainty 

and  most  control  subjects,  who  were  exposed  to  uncertainty  representation  during 
debriefing,  thought  that  the  representation  of  uncertainty  was  a  good  idea.  Subjects  were 
asked  to  recall  instances  in  which  uncertainty  representation  affected  their  decision.  Most 
of  them  were  unable  to  recall  more  than  one  or  two  decision  points.  They  thought, 
however,  that  representation  of  uncertainty  strengthens  the  awareness  to  the  limitations  of 
BMS  information  thereby  introducing  more  caution  into  the  way  commanders  use  BMS. 

It  should  be  noted  that  these  verbal  insights  were  not  supported  by  actual  performance. 
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As  we  have  seen,  less  additional  information  was  acquired  when  uncertainty  was 
displayed  than  when  it  was  not  displayed.  If  the  representation  of  uncertainty  does  indeed 
raise  the  awareness  to  gaps  between  BMS  information  and  reality  it  should  have  fostered 
requests  for  more  information  and  not  the  other  way  round. 

Some  subjects  suggested  that  uncertainty  should  be  displayed  as  a  layer  that  can  be 
turned  on  and  off  upon  request.  Other  subjects  thought  that  it  might  be  necessary  only  for 
beginners,  until  they  get  a  strong  handle  on  the  uncertain  nature  of  BMS.  Judging  by 
results,  at  least  10  out  of  the  16  SME  should  be  considered  as  "beginners,"  despite  their 
backgrounds  and  formal  experience. 

One  clear  disadvantage  of  uncertainty  representation  is  that  it  adds  more  information  to 
an  already  cluttered  display  (LeBlank  and  Summers,  2007).  Therefore,  as  some  subjects 
suggested,  uncertainty  information  should  be  a  selectable  (declutterable),  rather  than  a 
permanent,  layer  of  information. 

A  perspective  on  Sensemaking 

Most  decisions  made  during  the  above  research  were  not  optimal.  To  use  Klein  et  al. 
(2007)  terms,  they  did  not  do  a  very  good  "connecting  the  dots"  job.  Sensemaking  can  be 
defined  as  “exploiting  information  under  conditions  of  uncertainty,  complexity  and  time 
pressure  for  awareness,  understanding,  planning  and  decision  making.”  (Klein  et  al. 

2007).  Several  aspects  of  Sensemaking  are  related  to  team  work  and  are  not  relevant  to 
the  present  study;  nevertheless,  some  individual  aspects  of  Sensemaking  were  identified. 
On  the  individual  level,  most  failure  causes  defined  by  Klein  et  al.  (2007)  may  be 
identified  in  the  present  results: 

Effect  of  Mental  Models:  as  revealed  during  debriefing,  some  of  our  subjects  did  not  have 
a  clear  notion  in  regard  with  the  real  nature  and  value  of  BMS  information  and  tended  to 
perceive  what  they  see  as  what  really  exists  in  the  world.  In  other  words,  they  employed 
deficient  mental  models 

Confirmation  Bias:  (Actively  seeking  only  information  that  confirms  one’s  hypothesis 
even  in  the  face  of  disconfirming  evidence).  This  bias  could  not  be  identified  in  our  study 
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because  subjects  were  not  given  the  opportunity  to  select  between  alternative  sources  of 
information. 

Ignoring  discrepant  information:  this  kind  of  behavior  could  be  observed  in  the 
experimental  uncertainty  group  where  relevant  uncertainty  information  was  sometimes 
overlooked. 

Available  information  was  not  always  exploited:  some  subjects  tended  to  ignore 
uncertainty  representations  and  many  subjects  used  only  little  additional  information. 

Overconfidence  was  definitely  a  significant  factor.  Clearly  some  of  the  younger  and  less 
experienced  SMEs  perceived  the  situation  as  simpler  than  it  really  was,  misinterpreted 
several  of  the  situations  and  did  not  make  the  best  decisions. 

For  some  subjects,  the  uncertainty  representation  may  be  considered  as  "weak  signals".  In 
spite  of  the  emphasis  on  these  symbols  during  the  practice  trials  some  subjects  tended  not 
to  see  or  not  to  connect  these  "dots". 

It  would  be  interesting  to  investigate  the  possibility  of  improving  Sensemaking  with  the 
techniques  proposed  by  Klein  et  al.  (2007). 

Conclusions  and  Future  research 

The  main  purpose  of  the  present  research  was  to  investigate  the  usability  of  uncertainty 
representations  in  BMS  systems.  SMEs  expressed  strong  support  of  the  idea  of 
representing  uncertainty,  even  though  the  quantitative  results  showed  a  significant 
positive  effect  only  in  approximately  a  quarter  of  the  decision  points.  We  believe  that 
with  more  training  and  increased  understanding  of  BMS  capabilities  (and  limitations)  the 
advantages  of  uncertainty  information  may  increase.  Our  conclusion  is  that  such 
information  (when  available  and  reliable)  should  be  available  to  commanders  as  an 
optional  layer  of  information  on  the  BMS.  At  the  present  stage  we  do  not  think  that 
additional  research,  in  the  limited  context  of  objective  uncertainty  presented  herein,  may 
yield  much  deeper  insights. 

Another  important  finding  is  that  available  information  was  underused,  resulting  in  sub- 
optimal  decisions.  It  is  not  completely  clear  why  several  subjects  refrained  from  requiring 
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available  information.  Was  it  due  to  overconfidence,  or  perhaps  a  symptom  of  the 
syndrome  "men  do  not  ask  for  direction"?  Either  way,  this  kind  of  behavior  was 
counterproductive  and  ways  of  changing  it  should  be  devised. 

Some  individual  subjects  were  excellent  decision  makers  whereas  several  others  did  quite 
poorly.  A  cognitive  task  analysis  is  recommended  in  order  to  better  the  understanding  of 
the  basis  and  the  sources  of  good  decision  making  and  impart  it  with  the  less  proficient 
commanders. 

It  is  recommended  that  the  existing  infrastructure  of  the  IDF  BatLab  and  the  system 
developed  for  the  present  research  should  be  expanded  and  used  as  a  test-bed  for  team  SA 
and  Sensemaking  in  C3I  systems,  in  higher  level  operational  environments.  In  future 
stages  it  might  be  wise  to  adapt  Pfautz  et  al.  (2007)  insight  that  representing  uncertainty 
is  not  enough  and  that  it  is  necessary  to  adapt  a  comprehensive  meta-data  and  meta¬ 
information  approach. 

Some  potential  research  issues  that  have  already  been  identified  as  relevant  and  of 
possible  operational  benefit  include: 

•  Possible  advantages  of  using  3D  C3I  representations  as  compared  to  2D 
representation  (of  3D  information) 

•  Using  various  perspectives  of  terrain  representations  (e.g.,  top  down  view  versus 
panoramic  view,  flexible  views  controlled  by  the  operator,  etc.) 

•  Advantages  and  possible  drawbacks  of  combining  processed  intelligence  and 
tactical  information  with  raw  data  (e.g.,  live  or  recorded  video)  as  a  layer  on  a  C3I 
system  or  as  a  separate  source  of  information. 
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Appendix  1:  Scenario  script  and  decision  points 
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Outlined  on  the  map  (figure  2) 


The  following  Tables  outline  the  details  of  the  first  part  of  one  experimental  scenario  (out  of  eight),  including  one  decision  point  (out 
of  5)  and  one  SAGAT  Inquiry  (out  of  two  in  this  scenario). 
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separate  rules  and  is  not  triggered  by  scenario  events. 


Table  10:  Scenario  script  -  part  1 
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Appendix  2:  The  representation  of  uncertainty 

Uncertainty  in  general  is  represented  by  adding  a  semi-transparent  background  to  the  symbol.  The  examples 
below  are  all  on  the  infantry  tactical  symbol  but  may,  of  course  appear  with  any  relevant  tactical  symbol. 
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Appendix  3:  Simulation  and  Field  preparatory 
experiments 

Research  objectives 

The  Ground  Forces  of  the  IDF  are  testing  new  concepts  and  a  new  force  structure  for  low 
intensity  conflict  (LIC)  scenarios.  To  this  end  a  specialized  “Low  Intensity  Combat 
Tactical  Unit  (LIC  TU)”  has  been  conceived  and  designated  for  urban  terrain  warfare 
conflicts. 

The  LIC  TU  differs  from  other  ground  force  combat  units  in  three  main  aspects: 

a.  The  various  sub-units  composing  the  force  are  an  organic  part  of  the  unit  (e.g. 
engineering  forces,  armored  force). 

b.  The  unit  has  improved  information  management  abilities  based  on  a  common 
BMS.  The  BMS  provided  a  moving  map  display,  self  and  friendly  forces 
location  (based  on  GPS),  intelligence  (e.g.,  acquired  targets),  mission 
planning  capabilities  and  data  links. 

c.  Some  soldiers  have  autonomous  target  acquisition  devices  with  which  targets 
may  be  fed  into  the  BMS. 

The  structure  and  operational  concept  of  the  LIC  TU,  was  tested  in  two  experiments:  a 
Battle  Lab  experiment  (during  October,  2004)  and  a  field  experiment  (during  March, 
2005).  The  objective  of  the  experiments  was  to  consolidate  the  missions  and  roles  of  the 
LIC  TU  and  examine  various  aspects  of  the  new  unit:  its  organizational  structure, 
operational  methods,  the  use  of  newly  introduced  means  of  warfare  and  the  integrated 
battle  management  using  a  BMS. 

Researchers  roles 

In  both  experiments  PAMAM's  research  team  had  two  major  roles: 

a.  To  observe  and  test  the  research  notions  developed  during  the  pilot  study 
(Brickner  and  Lipshitz,  2004)  and  the  initial  phase  of  the  present  research 
program  (Brickner,  Sadot-Parag  and  Lipshitz,  2004). 

b.  Members  of  the  research  team  served  as  Human  Factors  specialists  for  the 
Israeli  Defense  Forces  (IDF)  presenting  their  conclusions  and 
recommendations  for  Human  Factors  improvements  and  changes  to  the  Israeli 
Ground  Forces. 

In  practice,  the  team  involvement  focused  on  the  following  aspects: 

□  Specify  the  "information  world"  of  commanders:  the  team  analyzed  the  information 
needs  and  requirements  of  different  commanding  levels  in  order  to  understand  their 
needs  and  provide  the  means  for  obtaining  high  levels  of  SA.  The  analysis  focused  on 
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the  necessary  adaptations  of  the  BMS  to  the  designated  LIC  TU  in  urban  battle 
activities. 

□  Uncertainty  representation:  The  ream  investigated  the  types  and  levels  of  uncertainty 
embedded  in  the  BMS;  pinpointed  gaps  between  real  world  situations  and  their 
representations  in  the  BMS  and  identified  their  significance. 

□  Attention  and  Workload:  The  research  team  evaluated  workload  and  resource 
allocation  of  commanders  during  various  missions  and  under  different  working 
methods.  The  relations  between  working  methods,  allocation  of  resources  and  team 
work  on  workload  were  assessed. 

□  Human  Factors  Engineering  aspects:  The  research  team  participated  in  the 
examination  of  the  usability  of  the  different  types  and  installations  of  the  BMS  and 
other  devices,  in  different  command  environments  (rear  and  front  command  posts; 
vehicle  and  soldier  mounted  system  operation. 

Experimental  methods 

Both  experiments  simulated  fighting  scenarios  in  urban  terrain  facing  terror  and  guerrilla 
forces.  The  subjects  were  IDF  service  men  and  women,  each  of  whom  played  his  /  her 
own  current  military  role.  In  both  experiments  participants  were  equipped  with  a  BMS 
and  supplementary  devices,  in  addition  to  their  personal  gear.  . 

The  Battle  Lab  experiment 

Approximately  50  subjects  participated  in  the  experiment,  representing  different  roles  and 
ranks  (infantry,  engineering  forces,  armored  vehicles,  snipers,  RPV  operators,  attack 
helicopter  pilots,  light  artillery  and  various  sensor  operators).  The  red  operators  were 
simulated  by  computer  models  with  optional  human  intervention.  Each  of  the  blue 
participants  viewed  an  accurate  3D  model  of  a  real  town  on  an  individual  display 
monitors,  from  their  simulated  position  in  the  world.  The  battle  space  could  be  viewed 
directly,  through  periscope,  IR  systems  or  light  intensifiers. 

Each  participant  was  equipped  with  a  BMS,  displayed  on  a  second  monitor,  a  GPS, 
speech  communication  devices,  various  acquisition  devices  and  different  weapon 
systems.  Blue  operators  could  move  by  foot  or  on  armored  vehicles;  each  operator  was 
able  to  acquire  targets  and  feed  them  into  the  BMS  for  the  common  use  of  all 
participants.  Blue  and  red  operators  were  represented  in  the  outside  world  scene  and 
could  see  each  other.  On  the  BMS  all  blue  forces  were  represented  as  tactical  symbols, 
whereas,  red  forces  were  represented  only  if  a  blue  force  acquired  them. 

The  scenario  generator  created  various  scenes  and  missions  that  were  prepared  by  Subject 
Matter  Experts  (SMEs).  The  participating  forces  were  required  to  plan  the  mission  and 
carry  it  out.  Performance  models  of  existing  systems  were  based  on  operational  analysis 
and  engineering  models;  and,  where  such  models  were  not  available,  task  analysis  and 
expert  opinions  were  used.  All  aspects  of  performance  were  recorded.  Debriefing  took 
place  at  the  end  of  each  operational  scenario  and  included  scenario  playback,  recognition 
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and  analysis  of  critical  events.  Qualitative  data  was  collected  by  trained  observers  and 
through  questionnaires 

The  Field  experiment 

An  active  infantry  battalion  supplemented  by  additional  forces  (armored  vehicles, 
engineering  forces,  snipers,  UAV  operators,  sensor  operators,  and  attack  helicopter 
pilots),  was  recruited  for  this  experiment,  which  took  place  in  an  urban  training  facility  of 
the  IDF.  The  training  area  included  a  small  model  of  an  urban  civilian  settlement  (one 
and  two-stories  houses,  mosque,  market  etc.),  sized  approximately  0.5  by  0.7  kilometer. 
Control  of  units’  parameters  and  actions  took  place  in  a  central  control  room  that 
gathered  all  recordable  experimental  information:  blue  and  red  forces  position,  fire 
engagements,  ammunition  inventory,  audio,  video  and  data  communications.  These 
recordings  produced  quantitative  performance  measures  (e.g.,  kill  efficiency  of  weapon 
systems).  Other,  qualitative  measures  were  obtained  through  observations  of  Subject- 
Matter-Experts  (SMEs)  and  through  interviews  and  questionnaires.  Observations  took 
place  throughout  all  mission  stages  (planning,  briefing,  execution  and  debriefing). 

Fire  engagements  were  recorded  using  a  MILES  system,  computerized  fire  models 
indicating  hits  and  misses  which  were  communicated  to  the  central  experimental  control 
room  which,  in  turn,  delivered  the  information  to  the  participating  forces  (e.g.,  who  was 
“injured”,  “killed”,  etc.). 

Commanders  of  the  blue  forces  were  equipped  with  a  mobile  BMS  device.  The  blue 
forces  were  able  to  observe  the  battlespace  directly  using  periscopes,  IR  systems  or  light 
intensifiers;  or  view  sensor  imagery.  Operators  were  able  to  designate  targets  and  acquire 
them  into  the  BMS,  operate  weapon  systems  and  perform  verbal  and  data 
communication. 

The  red  operators  were  simulated  by  an  enemy  staging  unit  that  played  the  roles  of  enemy 
forces  as  well  as  civilian  bystanders. 

The  battalion  engaged  in  a  series  of  five  operational  scenarios  developed  by  SMEs. 

Observations  and  relevant  Conclusions 

The  findings  described  below  were  compiled  from  both  experiments.  The  observations 
were  divided  along  the  basic  cognitive  concepts  on  which  the  research  is  based. 

Mental  Model 

BMS  design  must  match  commanders'  mental  models.  The  Integration  of  the  various 
information  sources  into  the  BMS  (e.g.  displaying  sensor  footprints  on  the  BMS  map),  is 
crucial  for  commanders’  success. 

Table  1 1  below  summarizes  some  of  the  main  observations  regarding  the  BMS  adequacy 
to  commanders’  critical  information  requirements  in  the  urban  battle  environment. 

The  information  is  sorted  based  on  the  analysis  of  the  “information  world”  of  each  key 
role.  The  example  below  refers  to  a  Battalion  or  Company  commander. 
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Table  11:  Required  and  display  information  on  a  commander’s  BMS  -  conclusions  and 
recommendations. 


Role 

Type  of 
Info. 

Required  info. 

Displayed 

info. 

Conclusion  & 
recommendations 

Battalion  / 

Company 

commander 

World 

Terrain  - 
relief 

Detailed  terrain, 

3D  info. 

Insufficient 

scaling 

Real  zoom-in  of  up  to 
1:1000 

Match  map  to 
terrain 

Inconsistencies, 
outdated  info. 

Updated  maps,  local 
updating  capabilities 

3D  terrain 
representation 

Only  symbolic 

3D  (contour 
lines,  shading) 

Test  the  usefulness  of 
perspective  3D 
displays 

Foliage  & 
human- 
made 
objects 

Key  object 
identifications 

Name  (e.g., 
roads)  & 

Number  (e.g., 
houses)  system 

Names  &  numbers 
invisible  at  small  scale 

Height  /  floors  of 
buildings 

None 

Display  as  label  or 
“tool  tip” 

Terrain  analysis 
on  relief  and 
human-made 
objects 

Partial 

Highly  necessary  for 
urban  activities. 

Friendly 

forces 

Tactical  symbols: 
Location,  status 

Location 

Display  status,  display 
minified  symbols  at 
small  scale. 

Indicate  known 
sources  of  uncertainty 

Direction  of 
viewing  & 
pointing 

Partial 

add 

Enemy 

forces 

Tactical  symbols: 
location,  status 

Location,  partial 

Indicate  known 
sources  of  uncertainty. 

Source  of 
information  (who 
acquired) 

Not  indicated 

Make  available  as  tool 
tip  or  pop-up 

Endurance  of 
acquired  targets 

5  minutes 

Should  vary  base  on 
target  type.  Should  be 
indicated,  e.g.,  fadeout. 

Civilians 

Gatherings, 
sensitive  locations 

Partially 
indicated,  often 
unknown. 

Required 

Team  and  Individual  SA  among  the  LIC  TU  commanders 

The  current  BMS  design  is  not  supportive  enough  of  the  commanders’  SA  and  the  shared 
SA  of  command  teams.  The  BMS  does  not  provide  a  sufficiently  complete  and  coherent 
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representation  of  the  current  situation.  Several  weak  points  affected  the  quality  of  the 
BMS,  for  example: 

•  Footprints  of  sensor  devices  were  not  displayed;  hence,  commander  had 
only  a  general  notion  of  available  sensor  information. 

®  Data  communication  was  slow  resulting  in  insufficiently  frequent  updates, 
causing  inaccurate  display  of  friend  and  foe  positions.  In  the  dense  urban 
area  these  relatively  small  location  errors  could  be  highly  significant. 

•  Target  acquisition  was  slow  and  partial;  commanders  devised  no  scheme 
for  effective  acquisition.  As  a  result,  there  was  much  uncertainty  regarding 
enemy  positions. 

•  Commanders  had  a  strong  inclination  to  rely  on  sensor  imagery  rather  than 
on  the  BMS,  consequently  they  tended  to  focus  on  specific  events  and 
neglect  the  big  picture. 

Despite  these  weaknesses  the  BMS  was  perceived  as  an  important  breakthrough.  In 
comparison  with  the  current  situation  in  which  communication  is  primarily  verbal,  the 
BMS  establishes  a  common  visual  language  that  significantly  enhances  individual  and 
team  SA. 

Attention  and  Workload 

In  current  operations  the  focus  of  attention  of  field  commanders  (up  to  the  brigade  level) 
is  on  the  real  world.  The  availability  of  a  BMS  and  multiple  sensor  imagery  creates  two 
types  of  issues.  First,  during  infantry  operations,  the  operator  must  carry  or  wear  all 
components  of  the  BMS;  this  may  affect  commanders’  mobility.  Secondly,  their  attention 
must  now  be  divided  between  several  old  and  new  sources.  This  re-raises  dilemmas  about 
to  role  and  desired  position  of  commanders;  should  commanders  be  physically  with  the 
force,  use  a  close  command  post  or  perhaps  operate  from  a  rear  command  center?  Each 
option  may  have  different  effects  on  workload  and  attention  allocation.  Our  conclusion  is 
that  the  commander  must  do  whatever  necessary  to  remain  on  top  of  things,  maintain  an 
overview  of  the  arena,  and  the  control  and  command  of  the  forces.  Hence,  he  /  she  should 
select  the  position  that  is  expected  to  yield  the  best  results.  Most  of  the  time,  the 
commander  should  engage  in  the  broader  picture  and  resist  the  temptation  to  focus  in 
real-time  detailed  (e.g.,  video  imagery).  Focusing  on  specific  actions  should  be  limited  to 
events  of  utmost  importance  were  the  overall  situation  is  controlled  by  another  team 
member  (e.g.,  deputy  commander). 

In  the  experiments,  some  incidents  of  excessive  workload  resulted  from  the  addition  of 
additional  forces  (armored  vehicles,  engineering  forces,  etc.).  The  demands  of  such  a 
multifaceted  force,  that  were  novel  to  the  otherwise  experienced  infantry  commander,  led 
to  inappropriate  teamwork.  The  commander  did  not  delegate  sufficient  authority  and 
sometimes  failed  to  allocate  tasks  and  responsibilities  to  his  team  members.  As  a  result, 
he  reached  peaks  of  overload  while  some  of  his  team  members  may  have  been  idle. 
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Uncertainty  within  the  BMS 

As  indicated  above  the  use  of  BMS  enhanced  SA  and  common  language.  Nevertheless, 
there  were  considerable  gaps  between  real-world  events  and  their  representation  on  the 
BMS,  i.e.,  there  were  significant  levels  of  uncertainty. 

Table  12  below  provides  a  summary  of  different  aspects  of  uncertain  information 
observed  in  the  BMS.  Uncertainty  is  analyzed  along  three  levels  of  SA  and  across  four 
dimensions:  completeness,  accuracy,  up-to-dateness  and  depth  (see  detailed  definitions  in 
Brickner  and  Lipshitz,  2004). 


Table  12:  Identified  layers  of  uncertainty  of  BMS  information 


Information 

world 

Dimension 

Uncertainty 

component 

Situation  Awareness  Level 

Perception 

Comprehension 

Projection 

Physical  World 
-  Terrain  relief 

Completeness 

Lack  of  height 
representation 

Partial  comprehension 
by  relief 

Partial  comprehension 

Accuracy 

Mismatch  between 
aerial  photograph 
and  terrain 

based  on  terrain 
analysis 

Lack  of  a 
comprehensive 
picture  reduces 
the  ability  to 

Up-to-dateness 

Mismatch  between 
aerial  photograph 
and  season  \  current 

state 

Lack  of  a 

comprehensive  terrain 
picture  due  to  an 
unclear  representation 
of  terrain 

project  future 
position 

Depth 

Physical  World 
-  Foliage  & 
human-made 
objects  (trees, 
houses) 

Completeness 

Accuracy 

No  representation 
of  elevation  / 
stories  /  height 
Terrain  analysis  did 
not  refers  to  foliage 
and  human-made 
objects 

Mismatch  between 
aerial  photograph 

Comprehending 
relative  height  by 
shading 

No  navigability 
comprehension 

Partial  comprehension 
of  structure  due  to 
lack  of  3D 

Lack  of  a 
comprehensive 
picture  reduces 
the  ability  to 

Up-to-dateness 

Depth 

Accuracy 

Up-to-dateness 

Depth 

and  features 

New  objects  not  in 
BMS.  Old  objects 
in  BMS  and  not 
outside 

representation 

project  future 
position 
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Information 

Uncertainty 

Situation  Awareness  Level 

world 

Dimension 

component 

Perception 

Comprehension 

Projection 

Friendly  forces 

Completeness 

Disappearance  of 
forces  inside 

buildings  was  not 
simulated  in 

No  direct 
representation  of 
predicted  future 
state  Limited 
projection 
ability  from 
planning  stage 
Slow  update 
rates  obstruct 
projection 

Accuracy 

experiments 

Location  accuracy 
limitations  due  to 
oversized  symbols 
Identification 
difficulties  of 
overlapping 
symbols 

no  representation  of 

Up-to-dateness 

low  due  to  slow 

status,  intention  etc. 

update  rates 

(Apart  from 

Depth 

extermination) 

Enemy  forces 

Completeness 

Comprehensive 
representation 
during  intelligence 
gathering  (prior  to 
making  contact) 

Very  partial  picture 
during  combat 

Partial  identification 
of  objects 

No  identification  of 
patterns  or  units  each 
object  treated 
independently 

No  projection 
capability 
beyond  general 
assessment  of 
alternative 
modes  of 
operation 

Accuracy 

Accuracy  depends 
on  acquiring  device 
Position  is  not 

Up-to-dateness 

dynamically 

updated 

No  representation  of 
position  inaccuracy 

Depth 

Source  (who 
acquired)  not 
identified 

No  representation  of 
Up-to-dateness  (e.g., 
time  of  acquisition) 

No  representation 
above  position  and 
possible  identification 
(capabilities,  status, 
intentions) 

Neutral  & 
Civilians 

Completeness 

Partial  and  general 
information  only 

Intuitive 

understanding 

Intuitive 

projection 

Accuracy 

most  information  is 

Up-to-dateness 

not  represented 

Depth 

The  BMS  may  create  a  misleading  illusion  of  complete  and  accurate  battlefield  SA.  It  is 
important  to  prevent  this  risk  without  crippling  the  very  use  of  the  BMS.  Adding 
uncertainty  representations  to  the  BMS  (e.g.,  time  of  last  update,  approximate\  estimated 
location)  may  reduce  the  risks  of  both  overconfidence  and  lack  of  confidence  in  the  BMS 
The  commander  who  is  aware  of  BMS  uncertainty  may  take  precautions  and  use 
additional  means  for  reducing  its  impact  (e.g.,  drill  down  for  more  information,  wait  for 
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the  completion  of  update  cycles,  use  voice  communication,  assess  the  reliability  of 
information  based  on  source,  etc.). 

The  projection  of  the  near  future  situation  of  friendly  forces  can  be  improved  by 
incorporating  planning-stage  information  into  the  BMS  display  (e.g.,  mission  stages).  In 
contrast  the  ability  to  project  enemy  actions  in  a  LIC  urban  scenario  is  practically 
nonexistent. 

Summary 

The  results  of  the  above  simulation  and  field  experiments  provided  valuable  insights  to 
some  of  the  issues  that  were  in  the  focal  point  of  the  current  research  program.  Important 
lessons  were  learned  about  gaps  between  commanders'  mental  models  and  BMS 
information  representation,  Sources  of  workload,  individual  and  team  SA.  Lessons 
learned  provided  insight  to  the  preparation  of  the  test-bed  of  the  present  simulation 
experiment.  Specifically,  real  sources  of  uncertainty  in  BMS  were  identified  and  some  of 
them  were  used  as  inputs  in  the  experiment. 
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Appendix  4:  The  Battle  Laboratory  (BatLab) 

Introduction 

The  Battle  Laboratory  (BatLab)  of  the  Israeli  Ground  Forces  is  a  simulation  laboratory 
that  is  capable  of  simulating  and  presenting  virtual  environments,  run  operational 
scenarios  and  allow  operators  to  perform  their  tasks  in  the  scenario  within  the  simulated 
environment  and  operate  a  variety  of  simulated  devices.  Events  as  well  as  performance 
measures  are  recorded  and  can  be  replayed  and  analyzed. 

BatLab  has  evolved  from  the  Human  Factors  Branch  in  the  early  90s  and  is  now  part  of  a 
center  for  military  research.  BatLab's  simulation  capabilities  are  based,  primarily,  on 
COTS  software  and  hardware. 

The  laboratory  focuses  on  human-machine  systems  and  has  three  primary  missions: 

*  To  test  and  specify  human-machine  systems  and  user  interface  designs 

■  To  test  operating  methods. 

■  To  test  Operational  Policies  and  the  operational  organization  of  fighting  units. 

During  its  first  years,  BatLab  focused  primarily  on  single  operators  and  small  groups,  in 
recent  years,  however,  technologies  (and  experience)  enable  BatLab  to  concurrently 
employ  approximately  50  positions.  This  enables  the  lab  to  simulate  and  test  the 
operation  of  much  larger  units  (e.g.  companies,  battalions,  command  posts,  etc.). 

Primary  features  of  BatLab 

1 .  Human  In  the  loop:  BatLab  investigates  functions  in  which  humans  play  a 
significant  part  as  operators  or  as  commanders. 

2.  Virtual  Environments:  Simulations  take  place  in  virtual  environments. 

3.  Real  time:  BatLab  simulates  operations  on  a  real  time  base.  Simulation  runs 
continuously  at  the  pace  of  the  regular  clock.  Only  marginal  events  (in  terms  of 
the  focus  of  research)  may  be  sped-up  to  save  time  (e.g.,  maintenance  times). 

4.  Demonstration  and  familiarization:  with  humans-in-the-loop,  real  pace 
simulation.  Because  it  is  not  feasible  to  perform  scores  of  runs  and  replications, 
the  lab  does  not  substitute  operational  analysis  simulation  tools.  In  the  preparation 
processes  of  experiments,  efforts  are  being  made  to  select  the  most  reasonable 
parameters,  thereby  reducing  variability.  In  some  cases  the  number  of  runs  may 
suffice  for  statistical  significance,  whereas,  in  other  cases,  results  remain 
qualitative. 

5.  Employment  and  integration  of  devices  and  weapon  systems:  the  simulation  of 
tested  devices  and  weapon  systems  operates  on  the  basis  of  the  same  simulation 
devices.  It  is,  therefore,  easy  to  introduce  new  devices  and  incorporate  them  into 
existing  simulation  facilities. 
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6.  Flexibility:  For  the  simulation  of  devices  and  weapon  system  it  is  sufficient  to 
have  a  good  specification  of  its  functionality  whereas  full  engineering 
specification  may  not  be  necessary.  Hence,  BatLab  can  test  systems  during  any 
phase  of  their  development  and  elaborate  the  simulation  as  the  system  develops. 

7.  Short  response  times:  the  time  required  for  simulation  is  a  fraction  of  the  time 
required  for  actual  development.  Given  the  necessary  definitions  specifications, 
the  lab  is  capable  of  rapid  preparation  and  execution  of  experiments. 

Components  of  BatLab 

The  basic  components  of  the  lab  are  presented  in  figure  2. 


Scenario  generators 

Red  Force 
Algorithms  and  /or 
human  operators 


Blue  Force 

Human  operators 


Experimental  control 
recording  &  processing 


Operational  control 
envelop 


Virtual  Theater 


Data  Bases 


DTM,  maps, 
orthophoto,  models... 


Input  from  subject 
matter  experts 

Operations  researclvv 
Engineering  research^/ 


J 


L 


Weapon  system 
simulation 


Weapon  system 
simulation 


Weapon  system 
simulation 


Figure  2:  Outline  of  the  basic  components  of  BatLab 

Theoretical  Foundations 

The  theoretical  foundations  for  most  BatLab  research  stem  from  two  main  sources. 
Whenever  possible,  existing  engineering  and  system  operation  models  are  applied  and 
adapted  as  necessary. 

If  such  models  do  not  yet  exist,  then  the  best  possible  approximation  is  created  with  the 
aid  of  individuals  and  teams  of  subject  matter  experts  (SME). 
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Outside  world  representation 

BatLab  has  the  necessary  component  for  the  simulation  of  the  world,  including:  maps, 
aerial  photographs,  orthophoto,  DTM,  models  of  various  systems,  platforms  and  terrain 
features,  battlefield  effects  and  atmospheric  effects.  These  enable  the  creation  of  3D 
representations  of  the  world  with  3D  objects  that  can  be  manipulated  in  various  ways 
(move,  shoot,  emit  smoke,  be  hit,  etc). 

Devices  and  weapon  systems 

The  simulation  ot  existing  systems  is  based  on  their  engineering  and  system  operation 
models,  whereas,  the  simulation  of  future  devices  is  based  on  SME  analyses,  including 
the  representation  of  user  interfaces. 

Communication  between  operators  comprises  speech  as  well  as  data  communication. 
Communication  features  may  include  - 

■  Communication  between  systems  (e.g..  and  aircraft  and  a  missile). 

■  Communication  between  command  and  control  systems  within  one  network  or 
between  different  networks. 

Scenario  generator 

The  scenario  generator  controls  all  components  that  are  not  controlled  by  humans  during 
the  experiment;  for  example,  the  red  forces  and  neighboring  blue  forces.  Hence,  the 
scenario  generator  is  responsible  for  important  components  of  the  "battlefield  theater" 
and  contributes  to  the  operators'  immersion  in  the  simulated  world. 

Monitoring  and  control 

Experiments  may  have  two  types  of  monitors  or  controllers: 

Experimental  monitoring:  the  technical  control  team  enables  the  smooth  performance  of 
simulation,  repetition  of  scenarios,  last  minute  changes,  etc. 

Operational  control:  The  operational  control  team  may  provide  additional  necessary 
interfaces  for  the  experimental  subjects.  For  example,  in  a  company  level  experiment  the 
battalion  command  may  be  represented  not  as  an  integral  part  of  the  experiment  but  as  an 
interface  that  add  realism  to  the  function  of  the  company  commander. 

Recording  and  processing 

All  events  in  a  simulation  are  recorded  and  can  be  replayed  and  reconstructed.  Statistical 
analysis  can  be  performed  on  the  data,  as  necessary. 
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